[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:05 dist-test-slave-c4w8 krb5kdc[32214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:05 dist-test-slave-c4w8 krb5kdc[32214](info): set up 2 sockets
Jun 20 08:04:05 dist-test-slave-c4w8 krb5kdc[32214](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 08:04:06.654325 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:06 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:06.785199 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:42901
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:42901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:06.875703 32227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:06.875986 32227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:06.876029 32227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:06.877395 32227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:06.877475 32227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:06.877501 32227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:06.877538 32227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:06.877561 32227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:06.879307 32227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:42901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:06.879633 32227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:06.879904 32227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:06.886948 32235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:06.887758 32233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:06.888353 32232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:06.888417 32227 server_base.cc:1034] running on GCE node
I20260620 08:04:06.888685 32227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:06.889005 32227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:06.890204 32227 hybrid_clock.cc:648] HybridClock initialized: now 1781942646890197 us; error 39 us; skew 500 ppm
Jun 20 08:04:06 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942646, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:06.892956 32227 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:06.893543 32227 webserver.cc:460] Webserver started at http://127.31.115.254:46853/ using document root <none> and password file <none>
I20260620 08:04:06.893889 32227 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:06.893976 32227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:06.894176 32227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:06.895787 32227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "c073c8c12cf34266b6a5cdd98aa3fc01"
format_stamp: "Formatted at 2026-06-20 08:04:06 on dist-test-slave-c4w8"
server_key: "8a878b02cc39c3c14e99b934f94e447e"
server_key_iv: "d9994d20db6b975a459a42292320d8fb"
server_key_version: "encryptionkey@0"
I20260620 08:04:06.896530 32227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "c073c8c12cf34266b6a5cdd98aa3fc01"
format_stamp: "Formatted at 2026-06-20 08:04:06 on dist-test-slave-c4w8"
server_key: "8a878b02cc39c3c14e99b934f94e447e"
server_key_iv: "d9994d20db6b975a459a42292320d8fb"
server_key_version: "encryptionkey@0"
I20260620 08:04:06.898433 32227 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 08:04:06.899483 32242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:06.899945 32227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:06.900089 32227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "c073c8c12cf34266b6a5cdd98aa3fc01"
format_stamp: "Formatted at 2026-06-20 08:04:06 on dist-test-slave-c4w8"
server_key: "8a878b02cc39c3c14e99b934f94e447e"
server_key_iv: "d9994d20db6b975a459a42292320d8fb"
server_key_version: "encryptionkey@0"
I20260620 08:04:06.900193 32227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:06.910900 32227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:06.913599 32227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:06.913837 32227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:06.918301 32227 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:42901
I20260620 08:04:06.918460 32294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:42901 every 8 connection(s)
I20260620 08:04:06.918800 32227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:06.919836 32295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:06.921378 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 32227
I20260620 08:04:06.921505 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:06.922334 32207 external_mini_cluster.cc:1428] Setting key a0ada128e613e9eb64b3931ed3646e54
I20260620 08:04:06.923462 32295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01: Bootstrap starting.
I20260620 08:04:06.924268 32295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:06.924611 32295 log.cc:824] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:06.925527 32295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01: No bootstrap required, opened a new log
I20260620 08:04:06.927721 32295 raft_consensus.cc:348] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42901 } }
I20260620 08:04:06.927933 32295 raft_consensus.cc:374] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:06.927983 32295 raft_consensus.cc:729] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c073c8c12cf34266b6a5cdd98aa3fc01, State: Initialized, Role: FOLLOWER
I20260620 08:04:06.928125 32295 consensus_queue.cc:260] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42901 } }
I20260620 08:04:06.928211 32295 raft_consensus.cc:388] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:06.928256 32295 raft_consensus.cc:482] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:06.928303 32295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:06.935384 32295 raft_consensus.cc:504] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42901 } }
Jun 20 08:04:06 dist-test-slave-c4w8 krb5kdc[32214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:06.935557 32295 leader_election.cc:302] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c073c8c12cf34266b6a5cdd98aa3fc01; no voters:
I20260620 08:04:06.935802 32295 leader_election.cc:288] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:06.935995 32300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:06.936161 32295 sys_catalog.cc:564] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:06.936194 32300 raft_consensus.cc:686] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [term 1 LEADER]: Becoming Leader. State: Replica: c073c8c12cf34266b6a5cdd98aa3fc01, State: Running, Role: LEADER
I20260620 08:04:06.936292 32300 consensus_queue.cc:237] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42901 } }
I20260620 08:04:06.936626 32301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42901 } } }
I20260620 08:04:06.936626 32302 sys_catalog.cc:455] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c073c8c12cf34266b6a5cdd98aa3fc01. Latest consensus state: current_term: 1 leader_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c073c8c12cf34266b6a5cdd98aa3fc01" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42901 } } }
I20260620 08:04:06.936796 32301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:06.936805 32302 sys_catalog.cc:458] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:06.937247 32307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:06.938158 32307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:06.940131 32307 catalog_manager.cc:1347] Generated new cluster ID: 15daf64c497a4dd9930a9e2ea8ecbdf0
I20260620 08:04:06.940189 32307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:06.941579 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:06.923482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.31.115.254:42901)
0620 08:04:06.923839 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:06.923851 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:06.924123 (+ 272us) server_negotiation.cc:407] Connection header received
0620 08:04:06.926928 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:06.926939 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:06.927004 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:06.927171 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:06.928987 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:06.930291 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:06.931499 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:06.931969 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:06.932071 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:06.936494 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:06.936499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:06.936499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:06.936512 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:06.939303 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:06.939905 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:06.939910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:06.939910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:06.940019 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:06.940400 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:06.940402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:06.940402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:06.940655 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:06.940820 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:06.941116 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 08:04:06.941261 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":232,"threads_started":1}
I20260620 08:04:06.959940 32307 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:06.960671 32307 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:06.966041 32307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c073c8c12cf34266b6a5cdd98aa3fc01: Generated new TSK 0
I20260620 08:04:06.966305 32307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:06.996989 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42901
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:07.083315 32321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:07.083559 32321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:07.083588 32321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:07.084780 32321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:07.084825 32321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:07.084856 32321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:07.086459 32321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:07.086676 32321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:07.086869 32321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:07.092338 32329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:07.092338 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:07.092362 32326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:07.092587 32321 server_base.cc:1034] running on GCE node
I20260620 08:04:07.092983 32321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:07.093309 32321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:07.094527 32321 hybrid_clock.cc:648] HybridClock initialized: now 1781942647094509 us; error 46 us; skew 500 ppm
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:07.097322 32321 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:07.097892 32321 webserver.cc:460] Webserver started at http://127.31.115.193:36671/ using document root <none> and password file <none>
I20260620 08:04:07.098133 32321 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:07.098203 32321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:07.098362 32321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:07.099468 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "a59a11c792dc46a292c812c81ebce283"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "577608938aa7d9082653832280191cf0"
server_key_iv: "a86a50182af7575c6e3f1376a401dd43"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.100189 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "a59a11c792dc46a292c812c81ebce283"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "577608938aa7d9082653832280191cf0"
server_key_iv: "a86a50182af7575c6e3f1376a401dd43"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.101773 32321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 08:04:07.102830 32336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.103071 32321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:07.103231 32321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "a59a11c792dc46a292c812c81ebce283"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "577608938aa7d9082653832280191cf0"
server_key_iv: "a86a50182af7575c6e3f1376a401dd43"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.103328 32321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:07.119577 32321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:07.122192 32321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:07.122396 32321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:07.122712 32321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:07.123117 32321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:07.123163 32321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.123196 32321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:07.123222 32321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.130031 32321 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:43365
I20260620 08:04:07.130119 32449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:43365 every 8 connection(s)
I20260620 08:04:07.130489 32321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:07.132305 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 32321
I20260620 08:04:07.132406 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:07.132611 32207 external_mini_cluster.cc:1428] Setting key 7d5c22b9a08df3220c79a908aa3336da
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:04:07.143491 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.131396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:55555 (local address 127.31.115.254:42901)
0620 08:04:07.131447 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:07.131450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:07.131712 (+ 262us) server_negotiation.cc:407] Connection header received
0620 08:04:07.132215 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:07.132216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:07.132231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:07.132318 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:07.133938 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.134818 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.136543 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.136780 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.136840 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.139679 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:07.139681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:07.139682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:07.139692 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:07.141702 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.142163 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.142165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.142166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.142239 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.142578 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.142580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.142580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.142738 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:07.142824 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:07.143323 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 08:04:07.143414 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:04:07.143628 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.131320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:42901 (local address 127.31.115.193:55555)
0620 08:04:07.131509 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:07.131558 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:07.132094 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:07.132468 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:07.132473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:07.133529 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:07.133832 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:07.133838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.134968 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.134969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:07.136435 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:07.136438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.136872 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.136874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:07.137035 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.137835 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:07.137847 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:07.139564 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:07.141806 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:07.141815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:07.141817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:07.142075 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:07.142349 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:07.142350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:07.142351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:07.142487 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:07.142846 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:07.142916 (+ 70us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:07.143113 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 08:04:07.143241 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 08:04:07.143399 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260620 08:04:07.144354 32450 heartbeater.cc:347] Connected to a master server at 127.31.115.254:42901
I20260620 08:04:07.144474 32450 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:07.144706 32450 heartbeater.cc:511] Master 127.31.115.254:42901 requested a full tablet report, sending...
I20260620 08:04:07.145412 32259 ts_manager.cc:194] Registered new tserver with Master: a59a11c792dc46a292c812c81ebce283 (127.31.115.193:43365)
I20260620 08:04:07.146246 32259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:55555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:07.164565 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42901
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:07.251693 32455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:07.251919 32455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:07.251945 32455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:07.253176 32455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:07.253226 32455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:07.253261 32455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:07.254894 32455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:07.255162 32455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:07.255432 32455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:07.260689 32461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:07.260689 32460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:07.260721 32463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:07.261045 32455 server_base.cc:1034] running on GCE node
I20260620 08:04:07.261322 32455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:07.261648 32455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:07.262845 32455 hybrid_clock.cc:648] HybridClock initialized: now 1781942647262825 us; error 35 us; skew 500 ppm
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:07.265587 32455 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:07.266140 32455 webserver.cc:460] Webserver started at http://127.31.115.194:37735/ using document root <none> and password file <none>
I20260620 08:04:07.266382 32455 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:07.266429 32455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:07.266561 32455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:07.267696 32455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "733a3a8d31004a76a355ddd2734f29f9"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "f8e6691eeb0a2a1d6c15ae137a2105ef"
server_key_iv: "fb02b684c406726b439bd9f7de4da9eb"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.268074 32455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "733a3a8d31004a76a355ddd2734f29f9"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "f8e6691eeb0a2a1d6c15ae137a2105ef"
server_key_iv: "fb02b684c406726b439bd9f7de4da9eb"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.269547 32455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:07.270536 32470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.270767 32455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:07.270864 32455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "733a3a8d31004a76a355ddd2734f29f9"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "f8e6691eeb0a2a1d6c15ae137a2105ef"
server_key_iv: "fb02b684c406726b439bd9f7de4da9eb"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.270944 32455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:07.286935 32455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:07.289358 32455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:07.289558 32455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:07.289893 32455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:07.290235 32455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:07.290271 32455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.290295 32455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:07.290307 32455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.296871 32455 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:45895
I20260620 08:04:07.296924 32583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:45895 every 8 connection(s)
I20260620 08:04:07.297284 32455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:07.300084 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 32455
I20260620 08:04:07.300222 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:07.300388 32207 external_mini_cluster.cc:1428] Setting key d2cc4334c1200037463f8439500b2fc5
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:07.309453 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.298205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:55119 (local address 127.31.115.254:42901)
0620 08:04:07.298306 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:07.298309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:07.298462 (+ 153us) server_negotiation.cc:407] Connection header received
0620 08:04:07.298960 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:07.298961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:07.298971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:07.299032 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:07.300349 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.300986 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.302246 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.302476 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.302575 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.305417 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:07.305419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:07.305419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:07.305429 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:07.307525 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.308174 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.308176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.308176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.308244 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.308643 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.308644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.308645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.308793 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:07.308867 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:07.309288 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 08:04:07.309373 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 08:04:07.309777 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.298158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:42901 (local address 127.31.115.194:55119)
0620 08:04:07.298320 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:07.298364 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:07.298869 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:07.299182 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:07.299187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:07.299944 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:07.300170 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:07.300176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.301145 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.301146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:07.302139 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:07.302142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.302594 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.302596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:07.302749 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.303378 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:07.303388 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:07.305304 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:07.307663 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:07.307674 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:07.307677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:07.308069 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:07.308332 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:07.308333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:07.308334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:07.308513 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:07.308890 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:07.308899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:07.309110 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 08:04:07.309247 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 08:04:07.309442 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260620 08:04:07.310431 32584 heartbeater.cc:347] Connected to a master server at 127.31.115.254:42901
I20260620 08:04:07.310602 32584 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:04:07.310875 32584 heartbeater.cc:511] Master 127.31.115.254:42901 requested a full tablet report, sending...
I20260620 08:04:07.311434 32259 ts_manager.cc:194] Registered new tserver with Master: 733a3a8d31004a76a355ddd2734f29f9 (127.31.115.194:45895)
I20260620 08:04:07.311995 32259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:55119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:07.332453 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42901
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:07.415729 32589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:07.415977 32589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:07.416018 32589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:07.417199 32589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:07.417269 32589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:07.417315 32589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:07.418864 32589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:07.419148 32589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:07.419389 32589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:07.425086 32597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:07.425107 32595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:07.425154 32594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:07.425321 32589 server_base.cc:1034] running on GCE node
I20260620 08:04:07.425537 32589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:07.425804 32589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:07.426972 32589 hybrid_clock.cc:648] HybridClock initialized: now 1781942647426933 us; error 51 us; skew 500 ppm
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:07.429797 32589 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:07.430311 32589 webserver.cc:460] Webserver started at http://127.31.115.195:34735/ using document root <none> and password file <none>
I20260620 08:04:07.430594 32589 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:07.430649 32589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:07.430751 32589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:07.431845 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "75896b6581244317a3253ea548927057"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "820cf1c651711be7241894743603a42a"
server_key_iv: "1f0dd503d4696d4cfd9cbe9a9f73c8ec"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.432185 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "75896b6581244317a3253ea548927057"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "820cf1c651711be7241894743603a42a"
server_key_iv: "1f0dd503d4696d4cfd9cbe9a9f73c8ec"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.433665 32589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:07.434477 32604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.434686 32589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:07.434774 32589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "75896b6581244317a3253ea548927057"
format_stamp: "Formatted at 2026-06-20 08:04:07 on dist-test-slave-c4w8"
server_key: "820cf1c651711be7241894743603a42a"
server_key_iv: "1f0dd503d4696d4cfd9cbe9a9f73c8ec"
server_key_version: "encryptionkey@0"
I20260620 08:04:07.434831 32589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:07.447866 32589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:07.450736 32589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:07.450922 32589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:07.451184 32589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:07.451615 32589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:07.451681 32589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.451714 32589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:07.451754 32589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:07.458878 32589 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:45791
I20260620 08:04:07.458976 32717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:45791 every 8 connection(s)
I20260620 08:04:07.459316 32589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:07.468042 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 32589
I20260620 08:04:07.468163 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:07.468336 32207 external_mini_cluster.cc:1428] Setting key a826dbec7b5b31cd0e32be5e1c298e00
I20260620 08:04:07.470323 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.460199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:41747 (local address 127.31.115.254:42901)
0620 08:04:07.460296 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:07.460298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:07.460448 (+ 150us) server_negotiation.cc:407] Connection header received
0620 08:04:07.460933 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:07.460935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:07.460945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:07.461016 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:07.462151 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.462677 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.463722 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.463932 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.463994 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.466660 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:07.466661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:07.466662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:07.466671 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:07.468601 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.469156 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.469157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.469158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.469227 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.469505 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.469506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.469507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.469748 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:07.469828 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:07.470125 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 08:04:07.470215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 08:04:07.470568 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.460149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:42901 (local address 127.31.115.195:41747)
0620 08:04:07.460316 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:07.460360 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:07.460846 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:07.461079 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:07.461083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:07.461824 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:07.462048 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:07.462055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.462793 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.462794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:07.463623 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:07.463625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.464032 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.464033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:07.464148 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.464798 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:07.464812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:07.466534 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:07.468694 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:07.468704 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:07.468707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:07.469058 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:07.469309 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:07.469310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:07.469310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:07.469433 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:07.469818 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:07.469826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:07.469983 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 08:04:07.470103 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 08:04:07.470286 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 08:04:07.471092 32718 heartbeater.cc:347] Connected to a master server at 127.31.115.254:42901
I20260620 08:04:07.471233 32718 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:07.471417 32718 heartbeater.cc:511] Master 127.31.115.254:42901 requested a full tablet report, sending...
I20260620 08:04:07.471837 32259 ts_manager.cc:194] Registered new tserver with Master: 75896b6581244317a3253ea548927057 (127.31.115.195:45791)
I20260620 08:04:07.472288 32259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:41747
I20260620 08:04:07.480950 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:07.505858 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.497597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.31.115.193:43365)
0620 08:04:07.497842 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:07.497847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:07.497861 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:04:07.497911 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:07.497913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:07.497927 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:07.497995 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:07.498546 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.499301 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.500058 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.500386 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.500443 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.502839 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:07.502848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:07.502848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:07.502856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:07.504531 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.504888 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.504891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.504891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.504948 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.505172 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.505173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.505173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.505337 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:07.505485 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:07.505597 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:04:07.505740 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":153,"threads_started":1}
W20260620 08:04:07.506414 32364 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56418
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:07.530349 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:07.522477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.31.115.193:43365)
0620 08:04:07.522576 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:07.522578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:07.522638 (+ 60us) server_negotiation.cc:407] Connection header received
0620 08:04:07.522690 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:07.522691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:07.522704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:07.522788 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:07.523428 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.523942 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.524665 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:07.524881 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:07.524940 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:07.527523 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:07.527525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:07.527526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:07.527533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:07.529006 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.529372 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.529373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.529373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.529426 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:07.529849 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:07.529850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:07.529851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:07.529966 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:07.530072 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:07.530173 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 08:04:07.530242 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 08:04:07.531001 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 32321
I20260620 08:04:07.536630 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 32455
I20260620 08:04:07.542802 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 32589
I20260620 08:04:07.547544 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 32227
2026-06-20T08:04:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1987 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32732](info): set up 2 sockets
Jun 20 08:04:07 dist-test-slave-c4w8 krb5kdc[32732](info): commencing operation
krb5kdc: starting...
W20260620 08:04:08.625810 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:08 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:08.746309 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:32887
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:08.830422 32745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:08.830720 32745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:08.830765 32745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:08.832010 32745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:08.832093 32745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:08.832118 32745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:08.832139 32745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:08.832170 32745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:08.833747 32745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:32887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:08.834048 32745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:08.834302 32745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:08.839501 32750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:08.839501 32751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:08.839501 32753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:08.839994 32745 server_base.cc:1034] running on GCE node
I20260620 08:04:08.840250 32745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:08.840761 32745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:08.841960 32745 hybrid_clock.cc:648] HybridClock initialized: now 1781942648841941 us; error 28 us; skew 500 ppm
Jun 20 08:04:08 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942648, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:08.844695 32745 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:08.845235 32745 webserver.cc:460] Webserver started at http://127.31.115.254:35589/ using document root <none> and password file <none>
I20260620 08:04:08.845504 32745 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:08.845551 32745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:08.845736 32745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:08.846854 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "3fe70b9e44af4af292c2db9be83c61f5"
format_stamp: "Formatted at 2026-06-20 08:04:08 on dist-test-slave-c4w8"
server_key: "795d2aa89a5b5903efc6b85492ce3750"
server_key_iv: "3d5ba31c183f3c97b1b16113b8626323"
server_key_version: "encryptionkey@0"
I20260620 08:04:08.847224 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "3fe70b9e44af4af292c2db9be83c61f5"
format_stamp: "Formatted at 2026-06-20 08:04:08 on dist-test-slave-c4w8"
server_key: "795d2aa89a5b5903efc6b85492ce3750"
server_key_iv: "3d5ba31c183f3c97b1b16113b8626323"
server_key_version: "encryptionkey@0"
I20260620 08:04:08.848677 32745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:08.849560 32760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:08.849856 32745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:08.849964 32745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "3fe70b9e44af4af292c2db9be83c61f5"
format_stamp: "Formatted at 2026-06-20 08:04:08 on dist-test-slave-c4w8"
server_key: "795d2aa89a5b5903efc6b85492ce3750"
server_key_iv: "3d5ba31c183f3c97b1b16113b8626323"
server_key_version: "encryptionkey@0"
I20260620 08:04:08.850050 32745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:08.859584 32745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:08.861907 32745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:08.862066 32745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:08.866421 32745 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:32887
I20260620 08:04:08.866460 344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:32887 every 8 connection(s)
I20260620 08:04:08.866832 32745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:08.867807 345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:08.870669 345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5: Bootstrap starting.
I20260620 08:04:08.871054 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 32745
I20260620 08:04:08.871194 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:08.871395 32207 external_mini_cluster.cc:1428] Setting key 53770082b0717329c5ec927eb8e41d7a
I20260620 08:04:08.871402 345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:08.871810 345 log.cc:824] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:08.872720 345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5: No bootstrap required, opened a new log
I20260620 08:04:08.875262 345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe70b9e44af4af292c2db9be83c61f5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32887 } }
I20260620 08:04:08.875442 345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:08.875473 345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe70b9e44af4af292c2db9be83c61f5, State: Initialized, Role: FOLLOWER
I20260620 08:04:08.875591 345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe70b9e44af4af292c2db9be83c61f5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32887 } }
I20260620 08:04:08.875661 345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:08.875691 345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:08.875747 345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:08.876354 345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe70b9e44af4af292c2db9be83c61f5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32887 } }
I20260620 08:04:08.876469 345 leader_election.cc:302] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fe70b9e44af4af292c2db9be83c61f5; no voters:
I20260620 08:04:08.876663 345 leader_election.cc:288] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:08.876724 350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:08.876878 350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe70b9e44af4af292c2db9be83c61f5, State: Running, Role: LEADER
I20260620 08:04:08.876963 345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:08.876963 350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe70b9e44af4af292c2db9be83c61f5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32887 } }
I20260620 08:04:08.877388 352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fe70b9e44af4af292c2db9be83c61f5. Latest consensus state: current_term: 1 leader_uuid: "3fe70b9e44af4af292c2db9be83c61f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe70b9e44af4af292c2db9be83c61f5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32887 } } }
I20260620 08:04:08.877482 352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:08.877698 351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fe70b9e44af4af292c2db9be83c61f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe70b9e44af4af292c2db9be83c61f5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32887 } } }
I20260620 08:04:08.877772 351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:08.877789 357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:08.878314 357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:08.879839 357 catalog_manager.cc:1347] Generated new cluster ID: 46571ede1343413792dfbdbed4569dda
I20260620 08:04:08.879889 357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:08 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:08.885219 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:08.872021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.31.115.254:32887)
0620 08:04:08.872220 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:08.872231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:08.872270 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:04:08.872769 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:08.872776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:08.872826 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:08.873017 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:08.873815 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:08.874726 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:08.875816 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:08.876253 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:08.876338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:08.881301 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:08.881305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:08.881305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:08.881318 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:08.883382 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:08.883784 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:08.883788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:08.883788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:08.883901 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:08.884107 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:08.884108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:08.884108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:08.884433 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:08.884605 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:08.884687 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 08:04:08.884831 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260620 08:04:08.888025 357 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:08.888725 357 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:08.904901 357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fe70b9e44af4af292c2db9be83c61f5: Generated new TSK 0
I20260620 08:04:08.905187 357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:08.941803 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32887
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:09.028187 371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:09.028439 371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:09.028484 371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:09.029803 371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:09.029874 371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:09.029919 371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:09.031544 371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:09.031854 371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:09.032120 371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:09.037670 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:09.037683 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:09.037670 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:09.037945 371 server_base.cc:1034] running on GCE node
I20260620 08:04:09.038192 371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:09.038465 371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:09.039623 371 hybrid_clock.cc:648] HybridClock initialized: now 1781942649039604 us; error 35 us; skew 500 ppm
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:09.042375 371 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:09.042981 371 webserver.cc:460] Webserver started at http://127.31.115.193:35059/ using document root <none> and password file <none>
I20260620 08:04:09.043243 371 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:09.043310 371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:09.043496 371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:09.044697 371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "3578dae98cb34ad0a5ad6718062a6f70"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "1bf5d8646ff6c0cd061ad4d5613d62ea"
server_key_iv: "b4a478d539d0fe757618843c98aa0b3b"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.045091 371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "3578dae98cb34ad0a5ad6718062a6f70"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "1bf5d8646ff6c0cd061ad4d5613d62ea"
server_key_iv: "b4a478d539d0fe757618843c98aa0b3b"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.046715 371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:09.047652 386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.047936 371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:09.048019 371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "3578dae98cb34ad0a5ad6718062a6f70"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "1bf5d8646ff6c0cd061ad4d5613d62ea"
server_key_iv: "b4a478d539d0fe757618843c98aa0b3b"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.048127 371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:09.099411 371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:09.101794 371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:09.101963 371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:09.102264 371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:09.102627 371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:09.102672 371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.102697 371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:09.102762 371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.109163 371 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:39545
I20260620 08:04:09.109232 499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:39545 every 8 connection(s)
I20260620 08:04:09.109566 371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:09.118178 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 371
I20260620 08:04:09.118301 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:09.118497 32207 external_mini_cluster.cc:1428] Setting key 31dff24e45dceae72c30feff4b1748c0
I20260620 08:04:09.120409 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.110362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:45959 (local address 127.31.115.254:32887)
0620 08:04:09.110433 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.110435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.110638 (+ 203us) server_negotiation.cc:407] Connection header received
0620 08:04:09.111721 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.111722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.111733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.111783 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.112440 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.113148 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.114303 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.114524 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.114587 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.117090 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.117091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.117092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.117098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.118834 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.119358 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.119360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.119360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.119433 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.119695 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.119696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.119697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.119832 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.119910 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.120213 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.120318 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:04:09.120821 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.110339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:32887 (local address 127.31.115.193:45959)
0620 08:04:09.110507 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.110547 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.111596 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.111854 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.111857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.112135 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:09.112347 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.112352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.113252 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.113253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.114209 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.114212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.114700 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.114701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.114813 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.115308 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:09.115317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:09.116957 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:09.118935 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:09.118942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:09.118944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:09.119266 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:09.119508 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:09.119509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:09.119509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:09.119627 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:09.119912 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:09.119921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:09.120070 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.120177 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.120345 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260620 08:04:09.121340 500 heartbeater.cc:347] Connected to a master server at 127.31.115.254:32887
I20260620 08:04:09.121438 500 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:09.121708 500 heartbeater.cc:511] Master 127.31.115.254:32887 requested a full tablet report, sending...
I20260620 08:04:09.122296 309 ts_manager.cc:194] Registered new tserver with Master: 3578dae98cb34ad0a5ad6718062a6f70 (127.31.115.193:39545)
I20260620 08:04:09.122910 309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:45959
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:09.145269 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32887
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:09.230532 505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:09.230800 505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:09.230845 505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:09.232101 505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:09.232182 505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:09.232239 505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:09.234009 505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:09.234306 505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:09.234546 505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:09.240182 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:09.240226 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:09.240262 505 server_base.cc:1034] running on GCE node
W20260620 08:04:09.240222 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:09.240609 505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:09.240904 505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:09.242071 505 hybrid_clock.cc:648] HybridClock initialized: now 1781942649242049 us; error 43 us; skew 500 ppm
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:09.245218 505 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:09.245833 505 webserver.cc:460] Webserver started at http://127.31.115.194:35301/ using document root <none> and password file <none>
I20260620 08:04:09.246093 505 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:09.246162 505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:09.246346 505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:09.247895 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "2fff1e4c37564c788cf869bf5ba31722"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "63e2e7c123159fad9f6d564e435bbf43"
server_key_iv: "f2ef71d031eef32f2f108546d8e220ca"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.248225 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "2fff1e4c37564c788cf869bf5ba31722"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "63e2e7c123159fad9f6d564e435bbf43"
server_key_iv: "f2ef71d031eef32f2f108546d8e220ca"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.249574 505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:09.250428 520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.250602 505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:09.250715 505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "2fff1e4c37564c788cf869bf5ba31722"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "63e2e7c123159fad9f6d564e435bbf43"
server_key_iv: "f2ef71d031eef32f2f108546d8e220ca"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.250794 505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:09.262609 505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:09.264792 505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:09.264967 505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:09.265218 505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:09.265750 505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:09.265854 505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.265900 505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:09.265929 505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.272336 505 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:34387
I20260620 08:04:09.272413 633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:34387 every 8 connection(s)
I20260620 08:04:09.272765 505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:09.280874 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 505
I20260620 08:04:09.280956 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:09.281169 32207 external_mini_cluster.cc:1428] Setting key 49c8cdeb093fb587b5477c6469719569
I20260620 08:04:09.282781 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.273563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:55479 (local address 127.31.115.254:32887)
0620 08:04:09.273658 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.273660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.273854 (+ 194us) server_negotiation.cc:407] Connection header received
0620 08:04:09.274817 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.274818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.274829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.274880 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.275473 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.275968 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.277090 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.277328 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.277411 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.279632 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.279633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.279634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.279642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.281217 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.281732 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.281733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.281734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.281794 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.282099 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.282100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.282100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.282220 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.282275 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.282632 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.282709 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:09.283188 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.273538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:32887 (local address 127.31.115.194:55479)
0620 08:04:09.273749 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.273787 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.274711 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.274940 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.274943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.275201 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:09.275404 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.275409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.276046 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.276047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.276939 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.276943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.277419 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.277420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.277582 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.278149 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:09.278157 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:09.279536 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:09.281318 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:09.281325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:09.281327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:09.281666 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:09.281883 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:09.281884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:09.281884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:09.282026 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:09.282288 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:09.282295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:09.282469 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.282580 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.282763 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":147,"threads_started":1}
I20260620 08:04:09.283797 634 heartbeater.cc:347] Connected to a master server at 127.31.115.254:32887
I20260620 08:04:09.283921 634 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:09.284102 634 heartbeater.cc:511] Master 127.31.115.254:32887 requested a full tablet report, sending...
I20260620 08:04:09.284479 309 ts_manager.cc:194] Registered new tserver with Master: 2fff1e4c37564c788cf869bf5ba31722 (127.31.115.194:34387)
I20260620 08:04:09.284816 309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:55479
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:09.308367 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32887
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:09.396066 639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:09.396325 639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:09.396369 639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:09.397737 639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:09.397799 639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:09.397851 639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:09.399536 639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:09.399827 639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:09.400082 639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:09.405695 644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:09.405704 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:09.405794 639 server_base.cc:1034] running on GCE node
W20260620 08:04:09.405696 645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:09.406173 639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:09.406379 639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:09.407578 639 hybrid_clock.cc:648] HybridClock initialized: now 1781942649407569 us; error 34 us; skew 500 ppm
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:09.410228 639 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:09.410753 639 webserver.cc:460] Webserver started at http://127.31.115.195:44365/ using document root <none> and password file <none>
I20260620 08:04:09.410995 639 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:09.411046 639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:09.411212 639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:09.412276 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "fdd07b404d6d4ced9d308935ec1ddc32"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "21713de46f3af9cdeab8d0789c055dd2"
server_key_iv: "a626997e3558c88c80257dce6ec5ebe2"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.412627 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "fdd07b404d6d4ced9d308935ec1ddc32"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "21713de46f3af9cdeab8d0789c055dd2"
server_key_iv: "a626997e3558c88c80257dce6ec5ebe2"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.414175 639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:09.415057 654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.415269 639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:09.415378 639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "fdd07b404d6d4ced9d308935ec1ddc32"
format_stamp: "Formatted at 2026-06-20 08:04:09 on dist-test-slave-c4w8"
server_key: "21713de46f3af9cdeab8d0789c055dd2"
server_key_iv: "a626997e3558c88c80257dce6ec5ebe2"
server_key_version: "encryptionkey@0"
I20260620 08:04:09.415452 639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:09.435252 639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:09.438052 639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:09.438268 639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:09.438544 639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:09.438895 639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:09.438952 639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.438997 639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:09.439025 639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:09.445726 639 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:34527
I20260620 08:04:09.445768 767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:34527 every 8 connection(s)
I20260620 08:04:09.446158 639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:09.453920 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 639
I20260620 08:04:09.454046 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:09.454226 32207 external_mini_cluster.cc:1428] Setting key 0b5b17ce4510d3e7c092fa52b62f77f8
I20260620 08:04:09.456931 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.446948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:35311 (local address 127.31.115.254:32887)
0620 08:04:09.447024 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.447026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.447265 (+ 239us) server_negotiation.cc:407] Connection header received
0620 08:04:09.448254 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.448255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.448267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.448329 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.448969 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.449480 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.450815 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.451025 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.451094 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.453734 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.453736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.453736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.453745 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.455513 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.455910 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.455911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.455911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.455967 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.456307 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.456308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.456308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.456420 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.456472 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.456796 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.456864 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:04:09.457181 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.446938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:32887 (local address 127.31.115.195:35311)
0620 08:04:09.447139 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.447178 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.448134 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.448408 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.448411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.448683 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:09.448881 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.448885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.449582 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.449583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.450711 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.450714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.451107 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.451108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.451213 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.451830 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:09.451840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:09.453569 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:09.455590 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:09.455598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:09.455601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:09.455840 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:09.456117 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:09.456117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:09.456118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:09.456240 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:09.456500 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:09.456507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:09.456646 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.456773 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.456924 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260620 08:04:09.457743 768 heartbeater.cc:347] Connected to a master server at 127.31.115.254:32887
I20260620 08:04:09.457864 768 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:09.458056 768 heartbeater.cc:511] Master 127.31.115.254:32887 requested a full tablet report, sending...
I20260620 08:04:09.458477 309 ts_manager.cc:194] Registered new tserver with Master: fdd07b404d6d4ced9d308935ec1ddc32 (127.31.115.195:34527)
I20260620 08:04:09.458909 309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:35311
I20260620 08:04:09.466614 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:04:09.474962 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.468405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.31.115.254:32887)
0620 08:04:09.468491 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.468495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.468583 (+ 88us) server_negotiation.cc:407] Connection header received
0620 08:04:09.468666 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.468667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.468678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.468726 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.469310 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.469878 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.470598 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.470798 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.470885 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.472142 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.472143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.472144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.472152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.473709 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.474108 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.474109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.474109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.474172 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.474378 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.474379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.474379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.474545 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.474655 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.474769 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.474854 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:04:09.477416 309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:09.479511 309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:09.486836 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.482557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:39545 (local address 127.0.0.1:50820)
0620 08:04:09.482759 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.482800 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.482865 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.483106 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.483109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.483112 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:09.483207 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.483208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.484406 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.484407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.485545 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.485547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.486515 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.486516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.486604 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.486619 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.486688 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.486746 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":146,"threads_started":1}
I20260620 08:04:09.486907 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.482557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34387 (local address 127.0.0.1:38220)
0620 08:04:09.482889 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.482902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.482959 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.483501 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.483502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.483504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:09.483556 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.483558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.484898 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.484899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.485891 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.485893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.486654 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.486654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.486713 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.486715 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.486770 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.486802 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2,"spinlock_wait_cycles":1280,"thread_start_us":66,"threads_started":1}
I20260620 08:04:09.487192 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.482699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.31.115.193:39545)
0620 08:04:09.482917 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.482923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.482934 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:09.482976 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.482978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.482995 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.483098 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:09.483317 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.484304 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.485660 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.486414 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.486516 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.486579 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.487047 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.487088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260620 08:04:09.487396 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.482812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:34527 (local address 127.0.0.1:34474)
0620 08:04:09.483551 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.483560 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.483615 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.483804 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.483805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.483806 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:09.483854 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.483856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.485041 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.485043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.486270 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.486273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.487171 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.487172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.487241 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.487246 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.487302 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.487339 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260620 08:04:09.487407 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.482697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.31.115.194:34387)
0620 08:04:09.483270 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.483273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.483283 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:09.483321 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.483323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.483338 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.483448 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:09.483655 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.484781 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.485980 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.486548 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.486651 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.486705 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.486785 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.486823 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":106,"threads_started":1}
I20260620 08:04:09.487757 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.482997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.31.115.195:34527)
0620 08:04:09.483199 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.483205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.483611 (+ 406us) server_negotiation.cc:407] Connection header received
0620 08:04:09.483681 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.483683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.483702 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.483786 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:09.483919 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.484932 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.486362 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.487086 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.487160 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.487206 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.487507 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.487670 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":121,"threads_started":1}
I20260620 08:04:09.487942 434 tablet_service.cc:1467] Processing CreateTablet for tablet 2c88ac3670a94907a44488a000d4c908 (DEFAULT_TABLE table=test-table [id=da77640be04341f4a0f695e63d929c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:09.487958 567 tablet_service.cc:1467] Processing CreateTablet for tablet 2c88ac3670a94907a44488a000d4c908 (DEFAULT_TABLE table=test-table [id=da77640be04341f4a0f695e63d929c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:09.488324 434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c88ac3670a94907a44488a000d4c908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:09.488324 567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c88ac3670a94907a44488a000d4c908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:09.488408 702 tablet_service.cc:1467] Processing CreateTablet for tablet 2c88ac3670a94907a44488a000d4c908 (DEFAULT_TABLE table=test-table [id=da77640be04341f4a0f695e63d929c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:09.488741 702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c88ac3670a94907a44488a000d4c908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:09.491775 785 tablet_bootstrap.cc:492] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: Bootstrap starting.
I20260620 08:04:09.492131 786 tablet_bootstrap.cc:492] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: Bootstrap starting.
I20260620 08:04:09.492513 785 tablet_bootstrap.cc:654] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:09.492794 786 tablet_bootstrap.cc:654] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:09.493185 785 log.cc:824] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:09.493364 786 log.cc:824] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:09.493763 787 tablet_bootstrap.cc:492] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: Bootstrap starting.
I20260620 08:04:09.494153 786 tablet_bootstrap.cc:492] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: No bootstrap required, opened a new log
I20260620 08:04:09.494161 785 tablet_bootstrap.cc:492] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: No bootstrap required, opened a new log
I20260620 08:04:09.494233 786 ts_tablet_manager.cc:1397] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:09.494233 785 ts_tablet_manager.cc:1397] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:04:09.494374 787 tablet_bootstrap.cc:654] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:09.494861 787 log.cc:824] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:09.495553 787 tablet_bootstrap.cc:492] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: No bootstrap required, opened a new log
I20260620 08:04:09.495648 787 ts_tablet_manager.cc:1397] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:04:09.496261 785 raft_consensus.cc:348] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.496255 786 raft_consensus.cc:348] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.496455 786 raft_consensus.cc:374] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:09.496455 785 raft_consensus.cc:374] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:09.496493 786 raft_consensus.cc:729] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fff1e4c37564c788cf869bf5ba31722, State: Initialized, Role: FOLLOWER
I20260620 08:04:09.496493 785 raft_consensus.cc:729] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd07b404d6d4ced9d308935ec1ddc32, State: Initialized, Role: FOLLOWER
I20260620 08:04:09.496646 785 consensus_queue.cc:260] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.496646 786 consensus_queue.cc:260] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.496953 785 ts_tablet_manager.cc:1428] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:09.496953 786 ts_tablet_manager.cc:1428] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:09.497076 634 heartbeater.cc:503] Master 127.31.115.254:32887 was elected leader, sending a full tablet report...
I20260620 08:04:09.497130 768 heartbeater.cc:503] Master 127.31.115.254:32887 was elected leader, sending a full tablet report...
I20260620 08:04:09.497464 787 raft_consensus.cc:348] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.497670 787 raft_consensus.cc:374] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:09.497709 787 raft_consensus.cc:729] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3578dae98cb34ad0a5ad6718062a6f70, State: Initialized, Role: FOLLOWER
I20260620 08:04:09.497817 787 consensus_queue.cc:260] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.498090 787 ts_tablet_manager.cc:1428] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 08:04:09.498253 500 heartbeater.cc:503] Master 127.31.115.254:32887 was elected leader, sending a full tablet report...
W20260620 08:04:09.523392 635 tablet.cc:2369] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:09.610368 501 tablet.cc:2369] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:09.696815 769 tablet.cc:2369] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:09.760885 791 raft_consensus.cc:482] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:09.761049 791 raft_consensus.cc:504] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.761579 791 leader_election.cc:288] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3578dae98cb34ad0a5ad6718062a6f70 (127.31.115.193:39545), 2fff1e4c37564c788cf869bf5ba31722 (127.31.115.194:34387)
I20260620 08:04:09.764778 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.761829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:39545 (local address 127.31.115.195:53379)
0620 08:04:09.761880 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.761893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.761940 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.762131 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.762132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.762134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:09.762237 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.762240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.763102 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.763104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.763908 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.763910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.764487 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.764489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.764580 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.764584 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.764644 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.764680 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260620 08:04:09.764824 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.761934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:53379 (local address 127.31.115.193:39545)
0620 08:04:09.761983 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.761985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.761993 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:09.762025 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.762026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.762039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.762100 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:09.762380 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.763005 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.764021 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.764397 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.764454 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.764462 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.764698 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.764749 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:04:09.764891 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.761838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34387 (local address 127.31.115.195:39083)
0620 08:04:09.762097 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:09.762106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:09.762173 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:09.762373 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:09.762374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:09.762376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:09.762448 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.762450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.763146 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.763147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.763926 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:09.763928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.764566 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.764567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:09.764675 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.764680 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:09.764753 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:04:09.764803 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":64,"thread_start_us":115,"threads_started":1}
I20260620 08:04:09.764896 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.761941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:39083 (local address 127.31.115.194:34387)
0620 08:04:09.762006 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.762008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.762174 (+ 166us) server_negotiation.cc:407] Connection header received
0620 08:04:09.762252 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.762253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.762270 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.762331 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:09.762566 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.763032 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.764067 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.764452 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.764543 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.764555 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.764772 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.764823 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:04:09.765246 588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c88ac3670a94907a44488a000d4c908" candidate_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fff1e4c37564c788cf869bf5ba31722" is_pre_election: true
I20260620 08:04:09.765259 454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c88ac3670a94907a44488a000d4c908" candidate_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3578dae98cb34ad0a5ad6718062a6f70" is_pre_election: true
I20260620 08:04:09.765436 588 raft_consensus.cc:2454] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd07b404d6d4ced9d308935ec1ddc32 in term 0.
I20260620 08:04:09.765447 454 raft_consensus.cc:2454] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd07b404d6d4ced9d308935ec1ddc32 in term 0.
I20260620 08:04:09.765730 656 leader_election.cc:302] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fff1e4c37564c788cf869bf5ba31722, fdd07b404d6d4ced9d308935ec1ddc32; no voters:
I20260620 08:04:09.765866 791 raft_consensus.cc:2781] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:09.765921 791 raft_consensus.cc:482] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:09.765954 791 raft_consensus.cc:3037] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:09.766670 791 raft_consensus.cc:504] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.766845 791 leader_election.cc:288] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [CANDIDATE]: Term 1 election: Requested vote from peers 3578dae98cb34ad0a5ad6718062a6f70 (127.31.115.193:39545), 2fff1e4c37564c788cf869bf5ba31722 (127.31.115.194:34387)
I20260620 08:04:09.766999 454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c88ac3670a94907a44488a000d4c908" candidate_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3578dae98cb34ad0a5ad6718062a6f70"
I20260620 08:04:09.767045 588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c88ac3670a94907a44488a000d4c908" candidate_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fff1e4c37564c788cf869bf5ba31722"
I20260620 08:04:09.767110 454 raft_consensus.cc:3037] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:09.767149 588 raft_consensus.cc:3037] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:09.767969 588 raft_consensus.cc:2454] T 2c88ac3670a94907a44488a000d4c908 P 2fff1e4c37564c788cf869bf5ba31722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd07b404d6d4ced9d308935ec1ddc32 in term 1.
I20260620 08:04:09.768148 454 raft_consensus.cc:2454] T 2c88ac3670a94907a44488a000d4c908 P 3578dae98cb34ad0a5ad6718062a6f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd07b404d6d4ced9d308935ec1ddc32 in term 1.
I20260620 08:04:09.768239 656 leader_election.cc:302] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fff1e4c37564c788cf869bf5ba31722, fdd07b404d6d4ced9d308935ec1ddc32; no voters:
I20260620 08:04:09.768373 791 raft_consensus.cc:2781] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:09.768533 791 raft_consensus.cc:686] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [term 1 LEADER]: Becoming Leader. State: Replica: fdd07b404d6d4ced9d308935ec1ddc32, State: Running, Role: LEADER
I20260620 08:04:09.768642 791 consensus_queue.cc:237] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } }
I20260620 08:04:09.769512 309 catalog_manager.cc:5495] T 2c88ac3670a94907a44488a000d4c908 P fdd07b404d6d4ced9d308935ec1ddc32 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdd07b404d6d4ced9d308935ec1ddc32 (127.31.115.195). New cstate: current_term: 1 leader_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3578dae98cb34ad0a5ad6718062a6f70" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fff1e4c37564c788cf869bf5ba31722" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd07b404d6d4ced9d308935ec1ddc32" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 34527 } health_report { overall_health: HEALTHY } } }
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:09.846933 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.838217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.31.115.193:39545)
0620 08:04:09.838299 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.838303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.838417 (+ 114us) server_negotiation.cc:407] Connection header received
0620 08:04:09.838451 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.838452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.838464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.838535 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.839133 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.839658 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.840446 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.840660 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.840727 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.843419 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.843434 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.843434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.843443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.845184 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.845917 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.845922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.845922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.846008 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.846225 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.846226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.846226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.846393 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.846640 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.846692 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.846766 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:09.873483 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.864628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.31.115.193:39545)
0620 08:04:09.864732 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.864734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.864745 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:09.864866 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.864868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.864879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.864942 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.865577 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.866158 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.867340 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.867727 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.867802 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.870505 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.870506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.870507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.870515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.872084 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.872474 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.872475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.872476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.872534 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.872798 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.872799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.872799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.872906 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.872984 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.873257 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.873332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:09.896773 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:09.889695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.31.115.193:39545)
0620 08:04:09.889785 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:09.889787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:09.889836 (+ 49us) server_negotiation.cc:407] Connection header received
0620 08:04:09.889902 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:09.889903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:09.889913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:09.889965 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:09.890537 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.891006 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.891726 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:09.891903 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:09.891953 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:09.894360 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:09.894361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:09.894361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:09.894369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:09.895686 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.896046 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.896047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.896047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.896125 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:09.896316 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:09.896316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:09.896317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:09.896415 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:09.896494 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:09.896640 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:04:09.896704 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 08:04:09.897349 414 tablet_service.cc:2908] Rejecting scan request for tablet 2c88ac3670a94907a44488a000d4c908: Uninitialized: clean time has not yet been initialized
I20260620 08:04:09.898342 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 371
I20260620 08:04:09.904458 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 505
I20260620 08:04:09.909430 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 639
I20260620 08:04:09.914414 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 32745
2026-06-20T08:04:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2366 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[809](info): set up 2 sockets
Jun 20 08:04:09 dist-test-slave-c4w8 krb5kdc[809](info): commencing operation
krb5kdc: starting...
W20260620 08:04:10.982364 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:11.109596 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:35567
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:11.192577 822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:11.192857 822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:11.192898 822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:11.194209 822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:11.194275 822 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:04:11.194298 822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:11.194321 822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:11.194350 822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:11.194378 822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:11.195911 822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:35567
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:11.196210 822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:11.196463 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:11.201435 827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.201552 828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.201556 830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:11.202261 822 server_base.cc:1034] running on GCE node
I20260620 08:04:11.202543 822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:11.202852 822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:11.204070 822 hybrid_clock.cc:648] HybridClock initialized: now 1781942651204054 us; error 29 us; skew 500 ppm
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:11.211340 822 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:11.211910 822 webserver.cc:460] Webserver started at http://127.31.115.254:36609/ using document root <none> and password file <none>
I20260620 08:04:11.212199 822 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:11.212275 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:11.212425 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:11.213555 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "b216ec759e97430080661291534bb2d6"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "7664408c6ef6819fa303ba5ebb938db2"
server_key_iv: "ce057e4e5cd1d63f0ad2280bee5d4080"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.213985 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "b216ec759e97430080661291534bb2d6"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "7664408c6ef6819fa303ba5ebb938db2"
server_key_iv: "ce057e4e5cd1d63f0ad2280bee5d4080"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.215446 822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:11.216384 837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.216640 822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:11.216765 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "b216ec759e97430080661291534bb2d6"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "7664408c6ef6819fa303ba5ebb938db2"
server_key_iv: "ce057e4e5cd1d63f0ad2280bee5d4080"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.216843 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:11.226964 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:11.229466 822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:11.229697 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:11.234532 822 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:35567
I20260620 08:04:11.234575 899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:35567 every 8 connection(s)
I20260620 08:04:11.234925 822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:11.235582 900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.238492 900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: Bootstrap starting.
I20260620 08:04:11.239218 900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.239552 900 log.cc:824] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:11.240350 900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: No bootstrap required, opened a new log
I20260620 08:04:11.242909 900 raft_consensus.cc:348] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ec759e97430080661291534bb2d6" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35567 } }
I20260620 08:04:11.243105 900 raft_consensus.cc:374] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.243156 900 raft_consensus.cc:729] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b216ec759e97430080661291534bb2d6, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.243368 900 consensus_queue.cc:260] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ec759e97430080661291534bb2d6" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35567 } }
I20260620 08:04:11.243463 900 raft_consensus.cc:388] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:11.243513 900 raft_consensus.cc:482] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:11.243573 900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:11.244247 900 raft_consensus.cc:504] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ec759e97430080661291534bb2d6" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35567 } }
I20260620 08:04:11.244379 900 leader_election.cc:302] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b216ec759e97430080661291534bb2d6; no voters:
I20260620 08:04:11.244602 900 leader_election.cc:288] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:11.244647 903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:11.244866 903 raft_consensus.cc:686] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [term 1 LEADER]: Becoming Leader. State: Replica: b216ec759e97430080661291534bb2d6, State: Running, Role: LEADER
I20260620 08:04:11.244951 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 822
I20260620 08:04:11.244962 903 consensus_queue.cc:237] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ec759e97430080661291534bb2d6" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35567 } }
I20260620 08:04:11.245019 900 sys_catalog.cc:564] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:11.245046 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:11.245249 32207 external_mini_cluster.cc:1428] Setting key 5c4e6aa644dcabb58929907491b9a798
I20260620 08:04:11.245400 905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b216ec759e97430080661291534bb2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ec759e97430080661291534bb2d6" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35567 } } }
I20260620 08:04:11.245398 904 sys_catalog.cc:455] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b216ec759e97430080661291534bb2d6. Latest consensus state: current_term: 1 leader_uuid: "b216ec759e97430080661291534bb2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ec759e97430080661291534bb2d6" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35567 } } }
I20260620 08:04:11.245507 905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:11.245508 904 sys_catalog.cc:458] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:11.245756 907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:11.246330 907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:11.247912 907 catalog_manager.cc:1347] Generated new cluster ID: f915551fbf2749af8f4a163d74b466d9
I20260620 08:04:11.247968 907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:11.257573 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.246119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.31.115.254:35567)
0620 08:04:11.246745 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.246757 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.246796 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:04:11.246981 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.246988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.247049 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.247224 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.247955 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.248953 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.249782 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.250210 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.250332 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.253214 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.253219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.253220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.253234 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.255896 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.256313 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.256317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.256317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.256421 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.256629 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.256630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.256630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.256905 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.257080 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.257182 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.257334 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":73,"threads_started":1}
I20260620 08:04:11.271327 907 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:11.271938 907 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:11.281224 907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: Generated new TSK 0
I20260620 08:04:11.281471 907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:11.316951 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:35567
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:11.417862 926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:11.418128 926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:11.418174 926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:11.419512 926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:11.419589 926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:11.419628 926 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:11.419663 926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:11.421283 926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:11.421572 926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:11.421851 926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:04:11.427173 926 server_base.cc:1034] running on GCE node
W20260620 08:04:11.427173 932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.427219 934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.427198 931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:11.427628 926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:11.427886 926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:11.429064 926 hybrid_clock.cc:648] HybridClock initialized: now 1781942651429060 us; error 37 us; skew 500 ppm
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:11.431691 926 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:11.432161 926 webserver.cc:460] Webserver started at http://127.31.115.193:37439/ using document root <none> and password file <none>
I20260620 08:04:11.432401 926 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:11.432472 926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:11.432627 926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:11.433812 926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "622d99cf77474717b99a87ac39225e94"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "d5f1c4eb66851344f59d1505bd5f2c39"
server_key_iv: "930158d80cf6e8b6ff1e2b97a8f51826"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.434201 926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "622d99cf77474717b99a87ac39225e94"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "d5f1c4eb66851344f59d1505bd5f2c39"
server_key_iv: "930158d80cf6e8b6ff1e2b97a8f51826"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.435662 926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:11.436542 941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.436807 926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:11.436926 926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "622d99cf77474717b99a87ac39225e94"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "d5f1c4eb66851344f59d1505bd5f2c39"
server_key_iv: "930158d80cf6e8b6ff1e2b97a8f51826"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.437011 926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:11.449524 926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:11.451687 926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:11.451874 926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:11.452545 926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:11.452610 926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.452648 926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:11.452670 926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.460346 926 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:40239
I20260620 08:04:11.460376 1056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:40239 every 8 connection(s)
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:11.460791 926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:11.462800 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 926
I20260620 08:04:11.462934 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:11.463124 32207 external_mini_cluster.cc:1428] Setting key ffdbeec14caf396edfb73f2f97750613
I20260620 08:04:11.464279 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.453003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:43157 (local address 127.31.115.254:35567)
0620 08:04:11.453081 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.453083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.453381 (+ 298us) server_negotiation.cc:407] Connection header received
0620 08:04:11.454177 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.454178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.454192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.454283 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.455270 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.455817 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.456923 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.457147 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.457222 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.461206 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.461208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.461208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.461221 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.462649 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.463210 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.463211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.463211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.463291 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.463567 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.463568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.463569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.463700 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.463776 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.464079 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.464188 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:04:11.464465 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.452993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.31.115.193:43157)
0620 08:04:11.453187 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.453246 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.454020 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.454385 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.454391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.454815 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:11.455153 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.455161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.455927 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.455930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.456815 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.456819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.457240 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.457241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.457365 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.457948 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:11.457957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:11.461099 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:11.462788 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.462797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.462800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.463113 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.463368 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.463369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.463370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.463494 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.463780 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:11.463788 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:11.463934 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.464041 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.464214 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260620 08:04:11.465806 1057 heartbeater.cc:347] Connected to a master server at 127.31.115.254:35567
I20260620 08:04:11.465914 1057 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:11.466133 1057 heartbeater.cc:511] Master 127.31.115.254:35567 requested a full tablet report, sending...
I20260620 08:04:11.466897 853 ts_manager.cc:194] Registered new tserver with Master: 622d99cf77474717b99a87ac39225e94 (127.31.115.193:40239)
I20260620 08:04:11.467878 853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:43157
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
I20260620 08:04:11.476493 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.470008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:53499 (local address 127.31.115.254:35567)
0620 08:04:11.470067 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.470069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.470160 (+ 91us) server_negotiation.cc:407] Connection header received
0620 08:04:11.470232 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.470233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.470242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.470290 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.470707 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.471191 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.472364 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.472613 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.472692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.473774 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.473775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.473776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.473790 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.475296 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.475661 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.475663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.475663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.475748 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.475962 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.475963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.475963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.476074 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.476151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.476309 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.476392 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:11.476495 1064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.469899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.31.115.193:53499)
0620 08:04:11.470074 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.470087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.470166 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.470300 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.470301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.470543 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:11.470612 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.470615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.471320 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.471321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.472267 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.472270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.472755 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.472756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.472830 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.473332 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:11.473337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:11.473646 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:11.475383 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.475384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.475385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.475587 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.475828 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.475829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.475830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.475898 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.476142 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:11.476143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:11.476192 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.476283 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.476393 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:11.493855 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:35567
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:11.580500 1066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:11.580717 1066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:11.580741 1066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:11.581971 1066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:11.582011 1066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:11.582032 1066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:11.582056 1066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:11.583504 1066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:11.583695 1066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:11.583875 1066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:11.588984 1072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.588994 1074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.589036 1071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:11.589133 1066 server_base.cc:1034] running on GCE node
I20260620 08:04:11.589398 1066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:11.589727 1066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:11.590900 1066 hybrid_clock.cc:648] HybridClock initialized: now 1781942651590885 us; error 42 us; skew 500 ppm
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:11.593420 1066 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:11.593927 1066 webserver.cc:460] Webserver started at http://127.31.115.194:37345/ using document root <none> and password file <none>
I20260620 08:04:11.594164 1066 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:11.594209 1066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:11.594342 1066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:11.595455 1066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "628aaa2ba0a64061aadf2d209b22af05"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "9de01f0436b8254710940d5e658b1745"
server_key_iv: "f2ed1c80fe32126d5d8cde05a8ceec70"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.595831 1066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "628aaa2ba0a64061aadf2d209b22af05"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "9de01f0436b8254710940d5e658b1745"
server_key_iv: "f2ed1c80fe32126d5d8cde05a8ceec70"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.597213 1066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:11.598115 1081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.598344 1066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:04:11.598420 1066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "628aaa2ba0a64061aadf2d209b22af05"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "9de01f0436b8254710940d5e658b1745"
server_key_iv: "f2ed1c80fe32126d5d8cde05a8ceec70"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.598507 1066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:11.613837 1066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:11.616015 1066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:11.616200 1066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:11.616765 1066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:11.616806 1066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.616832 1066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:11.616884 1066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.624083 1066 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:34581
I20260620 08:04:11.624148 1196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:34581 every 8 connection(s)
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:11.624455 1066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:11.627681 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.617328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:47863 (local address 127.31.115.254:35567)
0620 08:04:11.617406 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.617409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.617636 (+ 227us) server_negotiation.cc:407] Connection header received
0620 08:04:11.618175 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.618177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.618191 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.618282 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.619149 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.619989 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.621725 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.622075 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.622166 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.625084 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.625086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.625086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.625093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.626408 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.626856 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.626857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.626857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.626907 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.627125 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.627126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.627126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.627227 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.627277 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.627550 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.627610 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:04:11.627923 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.617298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.31.115.194:47863)
0620 08:04:11.617475 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.617517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.618064 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.618354 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.618359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.618701 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:11.619007 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.619015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.620087 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.620088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.621570 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.621575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.622206 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.622207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.622344 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.623024 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:11.623037 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:11.624983 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:11.626488 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.626495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.626504 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.626784 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.626968 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.626969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.626969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.627071 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.627305 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:11.627311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:11.627432 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.627525 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.627645 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260620 08:04:11.628508 1197 heartbeater.cc:347] Connected to a master server at 127.31.115.254:35567
I20260620 08:04:11.628602 1197 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:11.628813 1197 heartbeater.cc:511] Master 127.31.115.254:35567 requested a full tablet report, sending...
I20260620 08:04:11.629209 853 ts_manager.cc:194] Registered new tserver with Master: 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581)
I20260620 08:04:11.629585 853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:47863
I20260620 08:04:11.629748 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 1066
I20260620 08:04:11.629840 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:11.630038 32207 external_mini_cluster.cc:1428] Setting key b7ca352e1c920f6d3abe27744fa13d6f
I20260620 08:04:11.636155 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.629689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.31.115.194:32919)
0620 08:04:11.629817 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.629829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.629875 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.630428 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.630430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.630630 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:11.630735 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.630737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.631652 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.631652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.632227 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.632229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.632605 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.632605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.632687 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.633140 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:11.633144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:11.633414 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:11.635037 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.635038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.635039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.635220 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.635435 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.635436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.635436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.635494 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.635806 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:11.635807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:11.635857 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.635940 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.636047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260620 08:04:11.636157 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.630265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:32919 (local address 127.31.115.254:35567)
0620 08:04:11.630309 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.630311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.630318 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:11.630340 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.630341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.630351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.630406 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.630824 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.631559 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.632311 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.632520 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.632603 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.633499 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.633501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.633501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.633508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.634954 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.635306 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.635308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.635308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.635370 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.635567 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.635567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.635568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.635718 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.635815 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.635966 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.636044 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:11.657239 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:35567
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:11.743475 1206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:11.743745 1206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:11.743793 1206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:11.745078 1206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:11.745159 1206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:11.745204 1206 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:11.745236 1206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:11.747184 1206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:11.747485 1206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:11.747759 1206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:11.753151 1212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.753118 1211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:11.753118 1214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:11.753340 1206 server_base.cc:1034] running on GCE node
I20260620 08:04:11.753674 1206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:11.753895 1206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:11.755064 1206 hybrid_clock.cc:648] HybridClock initialized: now 1781942651755044 us; error 31 us; skew 500 ppm
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:11.757900 1206 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:11.758358 1206 webserver.cc:460] Webserver started at http://127.31.115.195:43239/ using document root <none> and password file <none>
I20260620 08:04:11.758620 1206 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:11.758687 1206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:11.758828 1206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:11.759920 1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "a874c913d59d41f3961e4a81b81d68bc"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "49a1c1d08324fcd9f149a16e869f2bce"
server_key_iv: "ffb583e79bb84f65d6bfcf10386fdb4e"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.760293 1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "a874c913d59d41f3961e4a81b81d68bc"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "49a1c1d08324fcd9f149a16e869f2bce"
server_key_iv: "ffb583e79bb84f65d6bfcf10386fdb4e"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.761723 1206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 08:04:11.762702 1221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.762962 1206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:11.763083 1206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "a874c913d59d41f3961e4a81b81d68bc"
format_stamp: "Formatted at 2026-06-20 08:04:11 on dist-test-slave-c4w8"
server_key: "49a1c1d08324fcd9f149a16e869f2bce"
server_key_iv: "ffb583e79bb84f65d6bfcf10386fdb4e"
server_key_version: "encryptionkey@0"
I20260620 08:04:11.763198 1206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:11.782882 1206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:11.785547 1206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:11.785763 1206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:11.786311 1206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:11.786373 1206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.786408 1206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:11.786465 1206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.793771 1206 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:43787
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:11.793851 1336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:43787 every 8 connection(s)
I20260620 08:04:11.794149 1206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:11.797330 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.787022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:34371 (local address 127.31.115.254:35567)
0620 08:04:11.787079 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.787082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.787352 (+ 270us) server_negotiation.cc:407] Connection header received
0620 08:04:11.788059 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.788061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.788075 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.788145 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.789120 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.790002 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.791186 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.791423 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.791520 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.794362 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.794364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.794364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.794373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.795899 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.796327 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.796329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.796329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.796385 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.796652 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.796653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.796653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.796779 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.796854 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.797151 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.797239 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:04:11.797500 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.787036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.31.115.195:34371)
0620 08:04:11.787220 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.787260 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.787939 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.788274 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.788280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.788689 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:11.789028 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.789035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.790131 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.790133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.791052 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.791055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.791548 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.791549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.791714 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.792336 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:11.792346 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:11.794262 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:11.796004 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.796011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.796013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.796235 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.796459 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.796460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.796460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.796575 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.796844 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:11.796851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:11.796995 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.797100 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.797264 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260620 08:04:11.798089 1337 heartbeater.cc:347] Connected to a master server at 127.31.115.254:35567
I20260620 08:04:11.798183 1337 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:11.798405 1337 heartbeater.cc:511] Master 127.31.115.254:35567 requested a full tablet report, sending...
I20260620 08:04:11.798815 853 ts_manager.cc:194] Registered new tserver with Master: a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787)
I20260620 08:04:11.799317 853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:34371
I20260620 08:04:11.802951 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 1206
I20260620 08:04:11.803035 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:11.803257 32207 external_mini_cluster.cc:1428] Setting key 638bebfaa90ed6f3db638b44acb501e4
I20260620 08:04:11.804284 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:04:11.804872 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.799137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:47869 (local address 127.31.115.254:35567)
0620 08:04:11.799178 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.799179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.799272 (+ 93us) server_negotiation.cc:407] Connection header received
0620 08:04:11.799299 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.799299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.799308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.799377 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.799745 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.800188 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.800842 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.801040 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.801116 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.802020 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.802021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.802021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.802027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.803635 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.804055 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.804056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.804056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.804121 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.804427 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.804428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.804428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.804539 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.804591 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.804756 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.804812 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:04:11.804992 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.799071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.31.115.195:47869)
0620 08:04:11.799197 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.799207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.799264 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.799395 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.799395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.799564 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:11.799642 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.799644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.800273 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.800274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.800766 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.800768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.801137 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.801138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.801230 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.801595 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:11.801599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:11.801942 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:11.803715 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.803716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.803717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.803985 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.804231 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:11.804232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:11.804249 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:11.804328 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:11.804605 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:11.804606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:11.804659 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.804742 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.804869 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:04:11 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:11.827402 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.820501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.31.115.254:35567)
0620 08:04:11.820547 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.820550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.820582 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:04:11.820685 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.820686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.820695 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.820746 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:11.821284 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.821916 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.822667 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.822872 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.822931 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.825085 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:11.825086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:11.825086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:11.825094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:11.826343 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.826639 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.826640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.826640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.826687 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:11.826916 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:11.826916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:11.826916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:11.827017 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:11.827092 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.827231 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.827294 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:11.829176 853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:11.830940 853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:11.838032 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.833678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.31.115.195:43787)
0620 08:04:11.833910 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.833921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.833946 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:04:11.833990 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.833992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.834007 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.834335 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:11.834393 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.835558 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.836847 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.837558 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.837661 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.837721 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.837887 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.837927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260620 08:04:11.838048 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.833472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:43787 (local address 127.0.0.1:35284)
0620 08:04:11.833586 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.833650 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.833760 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.834149 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.834153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.834155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:11.834248 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.834251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.835636 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.835636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.836754 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.836757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.837655 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.837656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.837764 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.837768 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.837865 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.837933 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":30,"thread_start_us":71,"threads_started":1}
I20260620 08:04:11.838049 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.833717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34581 (local address 127.0.0.1:51956)
0620 08:04:11.833984 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.833994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.834103 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.834433 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.834434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.834435 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:11.834480 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.834481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.835681 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.835682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.836651 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.836654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.837659 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.837660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.837741 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.837760 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.837865 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.837933 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260620 08:04:11.838600 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.834002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.31.115.194:34581)
0620 08:04:11.834230 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.834234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.834244 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:11.834280 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.834282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.834299 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.834371 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:11.834554 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.835619 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.836736 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.837534 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.837658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.837727 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.838163 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.838203 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260620 08:04:11.838919 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.833490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40239 (local address 127.0.0.1:42636)
0620 08:04:11.833660 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.833671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.833771 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.834464 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.834465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.834467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:11.834518 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.834520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.835443 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.835444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.836768 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.836770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.838699 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.838700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.838768 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.838773 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.838805 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.838847 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260620 08:04:11.838853 1271 tablet_service.cc:1467] Processing CreateTablet for tablet fcfbc389887b415abb3865d5eadf2688 (DEFAULT_TABLE table=test-table [id=ba964c2e766d4a84a6dbc4e05c232aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:11.839267 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfbc389887b415abb3865d5eadf2688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.839381 1131 tablet_service.cc:1467] Processing CreateTablet for tablet fcfbc389887b415abb3865d5eadf2688 (DEFAULT_TABLE table=test-table [id=ba964c2e766d4a84a6dbc4e05c232aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:11.839694 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfbc389887b415abb3865d5eadf2688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.842258 1360 tablet_bootstrap.cc:492] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Bootstrap starting.
I20260620 08:04:11.842828 1361 tablet_bootstrap.cc:492] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Bootstrap starting.
I20260620 08:04:11.843166 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.833880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.31.115.193:40239)
0620 08:04:11.834063 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.834068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.834079 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:11.834349 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.834352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.834370 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.834467 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:11.834602 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.835296 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.837760 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.838599 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.842883 (+ 4284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.842969 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.843003 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.843060 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260620 08:04:11.843425 1360 tablet_bootstrap.cc:654] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.843528 1361 tablet_bootstrap.cc:654] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.843729 1360 log.cc:824] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:11.843839 1361 log.cc:824] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:11.843924 991 tablet_service.cc:1467] Processing CreateTablet for tablet fcfbc389887b415abb3865d5eadf2688 (DEFAULT_TABLE table=test-table [id=ba964c2e766d4a84a6dbc4e05c232aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:11.844291 991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfbc389887b415abb3865d5eadf2688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.844576 1361 tablet_bootstrap.cc:492] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: No bootstrap required, opened a new log
I20260620 08:04:11.844592 1360 tablet_bootstrap.cc:492] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: No bootstrap required, opened a new log
I20260620 08:04:11.844652 1360 ts_tablet_manager.cc:1397] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:11.844648 1361 ts_tablet_manager.cc:1397] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:11.846395 1361 raft_consensus.cc:348] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.846518 1361 raft_consensus.cc:374] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.846539 1361 raft_consensus.cc:729] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628aaa2ba0a64061aadf2d209b22af05, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.846654 1361 consensus_queue.cc:260] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.846740 1360 raft_consensus.cc:348] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.846887 1360 raft_consensus.cc:374] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.846918 1360 raft_consensus.cc:729] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a874c913d59d41f3961e4a81b81d68bc, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.846923 1197 heartbeater.cc:503] Master 127.31.115.254:35567 was elected leader, sending a full tablet report...
I20260620 08:04:11.847041 1360 consensus_queue.cc:260] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.847194 1361 ts_tablet_manager.cc:1428] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:11.847338 1360 ts_tablet_manager.cc:1428] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:11.847477 1337 heartbeater.cc:503] Master 127.31.115.254:35567 was elected leader, sending a full tablet report...
I20260620 08:04:11.848038 1366 tablet_bootstrap.cc:492] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Bootstrap starting.
I20260620 08:04:11.848744 1366 tablet_bootstrap.cc:654] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.849146 1366 log.cc:824] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:11.850075 1366 tablet_bootstrap.cc:492] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: No bootstrap required, opened a new log
I20260620 08:04:11.850131 1366 ts_tablet_manager.cc:1397] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:11.851451 1366 raft_consensus.cc:348] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.851575 1366 raft_consensus.cc:374] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.851598 1366 raft_consensus.cc:729] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d99cf77474717b99a87ac39225e94, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.851675 1366 consensus_queue.cc:260] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.851886 1366 ts_tablet_manager.cc:1428] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:11.851958 1057 heartbeater.cc:503] Master 127.31.115.254:35567 was elected leader, sending a full tablet report...
I20260620 08:04:11.854040 1368 raft_consensus.cc:482] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:11.854147 1368 raft_consensus.cc:504] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.854475 1368 leader_election.cc:288] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787), 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581)
I20260620 08:04:11.857333 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.854848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:37863 (local address 127.31.115.194:34581)
0620 08:04:11.854890 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.854892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.854933 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:04:11.855001 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.855002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.855011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.855081 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:11.855261 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.855697 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.856624 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.856992 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.857047 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.857055 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.857220 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.857267 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:04:11.857342 1369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.854746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34581 (local address 127.31.115.193:37863)
0620 08:04:11.854874 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.854885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.854938 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.855083 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.855084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.855085 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:11.855197 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.855199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.855812 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.855813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.856520 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.856521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.857071 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.857072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.857145 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.857151 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.857197 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.857233 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260620 08:04:11.857416 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.854723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:44673 (local address 127.31.115.195:43787)
0620 08:04:11.854892 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.854894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.854903 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:11.854929 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.854930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.854939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.854993 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:11.855167 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.855675 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.856619 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.857005 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.857093 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.857101 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.857299 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.857342 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 08:04:11.857796 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.854632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:43787 (local address 127.31.115.193:44673)
0620 08:04:11.854696 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:11.854762 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:11.854811 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:11.855010 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:11.855011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:11.855013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:11.855087 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.855089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.855753 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.855754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.856518 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:11.856520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.857106 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.857107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:11.857220 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.857224 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:11.857279 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:04:11.857316 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260620 08:04:11.857793 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfbc389887b415abb3865d5eadf2688" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aaa2ba0a64061aadf2d209b22af05" is_pre_election: true
I20260620 08:04:11.857937 1151 raft_consensus.cc:2454] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 0.
I20260620 08:04:11.858130 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfbc389887b415abb3865d5eadf2688" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a874c913d59d41f3961e4a81b81d68bc" is_pre_election: true
I20260620 08:04:11.858144 942 leader_election.cc:302] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d99cf77474717b99a87ac39225e94, 628aaa2ba0a64061aadf2d209b22af05; no voters:
I20260620 08:04:11.858275 1291 raft_consensus.cc:2454] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 0.
I20260620 08:04:11.858296 1368 raft_consensus.cc:2781] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:11.858354 1368 raft_consensus.cc:482] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:11.858386 1368 raft_consensus.cc:3037] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:11.859272 1368 raft_consensus.cc:504] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.859404 1368 leader_election.cc:288] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 election: Requested vote from peers a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787), 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581)
I20260620 08:04:11.859581 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfbc389887b415abb3865d5eadf2688" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a874c913d59d41f3961e4a81b81d68bc"
I20260620 08:04:11.859637 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfbc389887b415abb3865d5eadf2688" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aaa2ba0a64061aadf2d209b22af05"
I20260620 08:04:11.859658 1291 raft_consensus.cc:3037] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:11.859745 1151 raft_consensus.cc:3037] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:11.860392 1291 raft_consensus.cc:2454] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 1.
I20260620 08:04:11.860461 1151 raft_consensus.cc:2454] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 1.
I20260620 08:04:11.860553 942 leader_election.cc:302] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d99cf77474717b99a87ac39225e94, a874c913d59d41f3961e4a81b81d68bc; no voters:
I20260620 08:04:11.860636 1368 raft_consensus.cc:2781] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:11.860778 1368 raft_consensus.cc:686] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 1 LEADER]: Becoming Leader. State: Replica: 622d99cf77474717b99a87ac39225e94, State: Running, Role: LEADER
I20260620 08:04:11.860875 1368 consensus_queue.cc:237] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } }
I20260620 08:04:11.861753 854 catalog_manager.cc:5495] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 622d99cf77474717b99a87ac39225e94 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:11.875010 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:11.872243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.31.115.193:40239)
0620 08:04:11.872309 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:11.872311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:11.872320 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:11.872454 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:11.872455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:11.872478 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:11.872545 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:11.872756 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.873226 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.874200 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:11.874392 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:11.874448 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:11.874669 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:11.874791 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:11.874851 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:11.874921 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 08:04:11.874940 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 08:04:11.875187 1198 tablet.cc:2369] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:11.877286 1151 raft_consensus.cc:1261] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Refusing update from remote peer 622d99cf77474717b99a87ac39225e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:11.877331 1291 raft_consensus.cc:1261] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Refusing update from remote peer 622d99cf77474717b99a87ac39225e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:11.877683 1370 consensus_queue.cc:1035] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:11.877808 1368 consensus_queue.cc:1035] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:11.881094 1375 mvcc.cc:205] Tried to move back new op lower bound from 7298837102086205440 to 7298837102022819840. Current Snapshot: MvccSnapshot[applied={T|T < 7298837102086205440}]
I20260620 08:04:11.881278 1374 mvcc.cc:205] Tried to move back new op lower bound from 7298837102086205440 to 7298837102022819840. Current Snapshot: MvccSnapshot[applied={T|T < 7298837102086205440 or (T in {7298837102086205440})}]
I20260620 08:04:11.884001 853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:11.884085 853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:11.884963 853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: Sending DeleteTablet for 3 replicas of tablet fcfbc389887b415abb3865d5eadf2688
I20260620 08:04:11.885290 1271 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfbc389887b415abb3865d5eadf2688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:11 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:35284
I20260620 08:04:11.885317 991 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfbc389887b415abb3865d5eadf2688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:11 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:42636
I20260620 08:04:11.885507 1131 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfbc389887b415abb3865d5eadf2688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:11 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:51956
I20260620 08:04:11.885880 1383 tablet_replica.cc:330] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: stopping tablet replica
I20260620 08:04:11.885989 1383 raft_consensus.cc:2229] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:11.885996 1385 tablet_replica.cc:330] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: stopping tablet replica
I20260620 08:04:11.886096 1383 raft_consensus.cc:2258] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:11.886085 853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:04:11.886154 1385 raft_consensus.cc:2229] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 08:04:11.886241 853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:11.886370 1385 raft_consensus.cc:2258] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:11.886562 1383 ts_tablet_manager.cc:1905] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:11.886823 1385 ts_tablet_manager.cc:1905] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:11.887713 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 8494f6d4c95449708a156b48a86f938d (DEFAULT_TABLE table=test-table [id=ebbd1e7c2b844e2aaf0338e4661249fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:11.887878 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8494f6d4c95449708a156b48a86f938d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.887991 1271 tablet_service.cc:1467] Processing CreateTablet for tablet 8494f6d4c95449708a156b48a86f938d (DEFAULT_TABLE table=test-table [id=ebbd1e7c2b844e2aaf0338e4661249fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:11.888006 991 tablet_service.cc:1467] Processing CreateTablet for tablet 8494f6d4c95449708a156b48a86f938d (DEFAULT_TABLE table=test-table [id=ebbd1e7c2b844e2aaf0338e4661249fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:11.888092 991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8494f6d4c95449708a156b48a86f938d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.888093 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8494f6d4c95449708a156b48a86f938d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:11.889281 1361 tablet_bootstrap.cc:492] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Bootstrap starting.
I20260620 08:04:11.889354 1360 tablet_bootstrap.cc:492] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Bootstrap starting.
I20260620 08:04:11.889756 1361 tablet_bootstrap.cc:654] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.889935 1360 tablet_bootstrap.cc:654] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.890105 1366 tablet_bootstrap.cc:492] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Bootstrap starting.
I20260620 08:04:11.890416 1385 ts_tablet_manager.cc:1918] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:11.890497 1385 log.cc:1196] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/fcfbc389887b415abb3865d5eadf2688
I20260620 08:04:11.890509 1366 tablet_bootstrap.cc:654] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:11.890616 1360 tablet_bootstrap.cc:492] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: No bootstrap required, opened a new log
I20260620 08:04:11.890668 1360 ts_tablet_manager.cc:1397] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:11.890784 1385 ts_tablet_manager.cc:1939] T fcfbc389887b415abb3865d5eadf2688 P 628aaa2ba0a64061aadf2d209b22af05: Deleting consensus metadata
I20260620 08:04:11.890833 1360 raft_consensus.cc:348] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:11.890924 1360 raft_consensus.cc:374] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.890954 1360 raft_consensus.cc:729] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a874c913d59d41f3961e4a81b81d68bc, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.890995 1360 consensus_queue.cc:260] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:11.891088 1360 ts_tablet_manager.cc:1428] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:11.891235 838 catalog_manager.cc:4841] TS 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581): tablet fcfbc389887b415abb3865d5eadf2688 (table test-table [id=ba964c2e766d4a84a6dbc4e05c232aa5]) successfully deleted
I20260620 08:04:11.891467 1383 ts_tablet_manager.cc:1918] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:11.891546 1383 log.cc:1196] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/fcfbc389887b415abb3865d5eadf2688
I20260620 08:04:11.891776 1366 tablet_bootstrap.cc:492] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: No bootstrap required, opened a new log
I20260620 08:04:11.891824 1383 ts_tablet_manager.cc:1939] T fcfbc389887b415abb3865d5eadf2688 P 622d99cf77474717b99a87ac39225e94: Deleting consensus metadata
I20260620 08:04:11.891825 1366 ts_tablet_manager.cc:1397] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:11.892108 1366 raft_consensus.cc:348] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:11.892189 1366 raft_consensus.cc:374] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.892211 1366 raft_consensus.cc:729] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d99cf77474717b99a87ac39225e94, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.892258 1366 consensus_queue.cc:260] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:11.892297 840 catalog_manager.cc:4841] TS 622d99cf77474717b99a87ac39225e94 (127.31.115.193:40239): tablet fcfbc389887b415abb3865d5eadf2688 (table test-table [id=ba964c2e766d4a84a6dbc4e05c232aa5]) successfully deleted
I20260620 08:04:11.892344 1366 ts_tablet_manager.cc:1428] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:11.892304 1384 tablet_replica.cc:330] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: stopping tablet replica
I20260620 08:04:11.892550 1384 raft_consensus.cc:2229] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:11.892658 1384 raft_consensus.cc:2258] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:11.893075 1361 tablet_bootstrap.cc:492] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: No bootstrap required, opened a new log
I20260620 08:04:11.893126 1361 ts_tablet_manager.cc:1397] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 08:04:11.893143 1384 ts_tablet_manager.cc:1905] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:11.893275 1361 raft_consensus.cc:348] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:11.893347 1361 raft_consensus.cc:374] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:11.893368 1361 raft_consensus.cc:729] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628aaa2ba0a64061aadf2d209b22af05, State: Initialized, Role: FOLLOWER
I20260620 08:04:11.893412 1361 consensus_queue.cc:260] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:11.893496 1361 ts_tablet_manager.cc:1428] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:11.894629 1384 ts_tablet_manager.cc:1918] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:11.894703 1384 log.cc:1196] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/fcfbc389887b415abb3865d5eadf2688
I20260620 08:04:11.894955 1384 ts_tablet_manager.cc:1939] T fcfbc389887b415abb3865d5eadf2688 P a874c913d59d41f3961e4a81b81d68bc: Deleting consensus metadata
I20260620 08:04:11.895366 838 catalog_manager.cc:4841] TS a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787): tablet fcfbc389887b415abb3865d5eadf2688 (table test-table [id=ba964c2e766d4a84a6dbc4e05c232aa5]) successfully deleted
W20260620 08:04:11.961795 1058 tablet.cc:2369] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:12.044778 1338 tablet.cc:2369] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:12.094470 1364 raft_consensus.cc:482] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:12.094575 1364 raft_consensus.cc:504] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.094915 1364 leader_election.cc:288] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622d99cf77474717b99a87ac39225e94 (127.31.115.193:40239), a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787)
I20260620 08:04:12.097815 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.095127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40239 (local address 127.31.115.194:41793)
0620 08:04:12.095171 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:12.095184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:12.095237 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:12.095438 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:12.095440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:12.095442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:12.095517 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.095518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.096196 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.096197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.096952 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.096954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.097507 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.097508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.097592 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.097595 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:12.097700 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:04:12.097739 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 08:04:12.097815 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.095213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:41793 (local address 127.31.115.193:40239)
0620 08:04:12.095275 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.095277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.095294 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:04:12.095325 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.095325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.095333 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.095416 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:12.095592 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.096077 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.097048 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.097414 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.097509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.097518 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.097713 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.097754 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:04:12.098098 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.095219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:53685 (local address 127.31.115.195:43787)
0620 08:04:12.095282 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.095284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.095493 (+ 209us) server_negotiation.cc:407] Connection header received
0620 08:04:12.095555 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.095557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.095566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.095636 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:12.095812 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.096272 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.097203 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.097648 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.097730 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.097738 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.097935 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.098001 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:12.098119 1011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8494f6d4c95449708a156b48a86f938d" candidate_uuid: "628aaa2ba0a64061aadf2d209b22af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d99cf77474717b99a87ac39225e94" is_pre_election: true
I20260620 08:04:12.098256 1011 raft_consensus.cc:2454] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628aaa2ba0a64061aadf2d209b22af05 in term 0.
I20260620 08:04:12.098416 1389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.095127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:43787 (local address 127.31.115.194:53685)
0620 08:04:12.095415 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:12.095424 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:12.095503 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:12.095637 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:12.095638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:12.095640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:12.095731 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.095733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.096369 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.096370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.097112 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.097114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.097760 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.097762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.097828 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.097831 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:12.098268 (+ 437us) client_negotiation.cc:240] Negotiation successful
0620 08:04:12.098314 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":162,"thread_start_us":90,"threads_started":1}
I20260620 08:04:12.098511 1084 leader_election.cc:302] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d99cf77474717b99a87ac39225e94, 628aaa2ba0a64061aadf2d209b22af05; no voters:
I20260620 08:04:12.098644 1364 raft_consensus.cc:2781] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:12.098687 1364 raft_consensus.cc:482] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:12.098644 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8494f6d4c95449708a156b48a86f938d" candidate_uuid: "628aaa2ba0a64061aadf2d209b22af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a874c913d59d41f3961e4a81b81d68bc" is_pre_election: true
I20260620 08:04:12.098714 1364 raft_consensus.cc:3037] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:12.098752 1291 raft_consensus.cc:2454] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628aaa2ba0a64061aadf2d209b22af05 in term 0.
I20260620 08:04:12.099417 1364 raft_consensus.cc:504] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.099568 1364 leader_election.cc:288] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [CANDIDATE]: Term 1 election: Requested vote from peers 622d99cf77474717b99a87ac39225e94 (127.31.115.193:40239), a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787)
I20260620 08:04:12.099695 1011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8494f6d4c95449708a156b48a86f938d" candidate_uuid: "628aaa2ba0a64061aadf2d209b22af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622d99cf77474717b99a87ac39225e94"
I20260620 08:04:12.099730 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8494f6d4c95449708a156b48a86f938d" candidate_uuid: "628aaa2ba0a64061aadf2d209b22af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a874c913d59d41f3961e4a81b81d68bc"
I20260620 08:04:12.099772 1011 raft_consensus.cc:3037] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:12.099792 1291 raft_consensus.cc:3037] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:12.100528 1291 raft_consensus.cc:2454] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628aaa2ba0a64061aadf2d209b22af05 in term 1.
I20260620 08:04:12.100566 1011 raft_consensus.cc:2454] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628aaa2ba0a64061aadf2d209b22af05 in term 1.
I20260620 08:04:12.100733 1082 leader_election.cc:302] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 622d99cf77474717b99a87ac39225e94, 628aaa2ba0a64061aadf2d209b22af05, a874c913d59d41f3961e4a81b81d68bc; no voters:
I20260620 08:04:12.100822 1364 raft_consensus.cc:2781] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:12.100937 1364 raft_consensus.cc:686] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 1 LEADER]: Becoming Leader. State: Replica: 628aaa2ba0a64061aadf2d209b22af05, State: Running, Role: LEADER
I20260620 08:04:12.101006 1364 consensus_queue.cc:237] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.101763 853 catalog_manager.cc:5495] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "628aaa2ba0a64061aadf2d209b22af05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } health_report { overall_health: UNKNOWN } } }
Jun 20 08:04:12 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:12.146700 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.138410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35567 (local address 127.0.0.1:59686)
0620 08:04:12.138578 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:12.138595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:12.138658 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:12.138954 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:12.138955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:12.139337 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:12.139398 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.139401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.140278 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.140279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.140941 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.140944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.141300 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.141301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.141350 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.142106 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:12.142111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:12.143609 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:12.145397 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:12.145405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:12.145408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:12.145703 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:12.145919 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:12.145920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:12.145921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:12.145997 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:12.146362 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:12.146369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:12.146418 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 08:04:12.146486 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:04:12.146592 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260620 08:04:12.146893 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.138521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.31.115.254:35567)
0620 08:04:12.138655 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.138657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.138667 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:12.138831 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.138832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.138847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.138916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:12.139502 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.140175 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.141036 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.141227 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.141284 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.143697 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:12.143698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:12.143698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:12.143705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:12.145317 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:12.145784 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:12.145785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:12.145785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:12.145837 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:12.146124 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:12.146126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:12.146126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:12.146257 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:12.146339 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.146547 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.146646 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":65}
I20260620 08:04:12.149297 853 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59686:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:04:12.149501 853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:12.151260 991 tablet_service.cc:1467] Processing CreateTablet for tablet c050b661030f4deea913abec8f90d5f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0010c21630904c7988cbc0c3910b15f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:12.151260 1131 tablet_service.cc:1467] Processing CreateTablet for tablet c050b661030f4deea913abec8f90d5f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0010c21630904c7988cbc0c3910b15f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:12.151376 1271 tablet_service.cc:1467] Processing CreateTablet for tablet c050b661030f4deea913abec8f90d5f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0010c21630904c7988cbc0c3910b15f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:12.151432 991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c050b661030f4deea913abec8f90d5f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:12.151432 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c050b661030f4deea913abec8f90d5f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:12.151535 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c050b661030f4deea913abec8f90d5f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:12.152879 1360 tablet_bootstrap.cc:492] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: Bootstrap starting.
I20260620 08:04:12.152979 1366 tablet_bootstrap.cc:492] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: Bootstrap starting.
I20260620 08:04:12.152989 1361 tablet_bootstrap.cc:492] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: Bootstrap starting.
I20260620 08:04:12.153288 1366 tablet_bootstrap.cc:654] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:12.153373 1361 tablet_bootstrap.cc:654] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:12.153373 1360 tablet_bootstrap.cc:654] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:12.153911 1366 tablet_bootstrap.cc:492] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: No bootstrap required, opened a new log
I20260620 08:04:12.153972 1366 ts_tablet_manager.cc:1397] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:12.154099 1366 raft_consensus.cc:348] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.154178 1366 raft_consensus.cc:374] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:12.154193 1366 raft_consensus.cc:729] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622d99cf77474717b99a87ac39225e94, State: Initialized, Role: FOLLOWER
I20260620 08:04:12.154206 1361 tablet_bootstrap.cc:492] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: No bootstrap required, opened a new log
I20260620 08:04:12.154258 1361 ts_tablet_manager.cc:1397] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:12.154251 1366 consensus_queue.cc:260] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.154345 1366 ts_tablet_manager.cc:1428] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:12.154371 1380 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.154400 1361 raft_consensus.cc:348] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.154450 1380 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.154476 1361 raft_consensus.cc:374] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:12.154496 1361 raft_consensus.cc:729] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628aaa2ba0a64061aadf2d209b22af05, State: Initialized, Role: FOLLOWER
I20260620 08:04:12.154546 1361 consensus_queue.cc:260] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.154656 1361 ts_tablet_manager.cc:1428] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:12.154742 1360 tablet_bootstrap.cc:492] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: No bootstrap required, opened a new log
I20260620 08:04:12.154701 1364 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.154780 1364 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.154790 1360 ts_tablet_manager.cc:1397] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:04:12.154942 1360 raft_consensus.cc:348] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.155026 1360 raft_consensus.cc:374] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:12.155056 1360 raft_consensus.cc:729] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a874c913d59d41f3961e4a81b81d68bc, State: Initialized, Role: FOLLOWER
I20260620 08:04:12.155099 1360 consensus_queue.cc:260] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.155215 1360 ts_tablet_manager.cc:1428] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:12.155308 1365 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.155432 1365 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.228549 1380 raft_consensus.cc:482] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:12.228657 1380 raft_consensus.cc:504] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.228866 1380 leader_election.cc:288] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581), a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787)
I20260620 08:04:12.229108 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c050b661030f4deea913abec8f90d5f2" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aaa2ba0a64061aadf2d209b22af05" is_pre_election: true
I20260620 08:04:12.229228 1151 raft_consensus.cc:2454] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 0.
I20260620 08:04:12.229195 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c050b661030f4deea913abec8f90d5f2" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a874c913d59d41f3961e4a81b81d68bc" is_pre_election: true
I20260620 08:04:12.229352 1291 raft_consensus.cc:2454] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 0.
I20260620 08:04:12.229398 942 leader_election.cc:302] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d99cf77474717b99a87ac39225e94, 628aaa2ba0a64061aadf2d209b22af05; no voters:
I20260620 08:04:12.229560 1380 raft_consensus.cc:2781] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:12.229687 1380 raft_consensus.cc:482] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:12.229722 1380 raft_consensus.cc:3037] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:12.230453 1380 raft_consensus.cc:504] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.230607 1380 leader_election.cc:288] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 election: Requested vote from peers 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581), a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787)
I20260620 08:04:12.230809 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c050b661030f4deea913abec8f90d5f2" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628aaa2ba0a64061aadf2d209b22af05"
I20260620 08:04:12.230823 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c050b661030f4deea913abec8f90d5f2" candidate_uuid: "622d99cf77474717b99a87ac39225e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a874c913d59d41f3961e4a81b81d68bc"
I20260620 08:04:12.230973 1291 raft_consensus.cc:3037] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:12.230973 1151 raft_consensus.cc:3037] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:12.231732 1151 raft_consensus.cc:2454] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 1.
I20260620 08:04:12.231732 1291 raft_consensus.cc:2454] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 622d99cf77474717b99a87ac39225e94 in term 1.
I20260620 08:04:12.231969 942 leader_election.cc:302] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622d99cf77474717b99a87ac39225e94, a874c913d59d41f3961e4a81b81d68bc; no voters:
I20260620 08:04:12.232091 1380 raft_consensus.cc:2781] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:12.232172 1380 raft_consensus.cc:686] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [term 1 LEADER]: Becoming Leader. State: Replica: 622d99cf77474717b99a87ac39225e94, State: Running, Role: LEADER
I20260620 08:04:12.232249 1380 consensus_queue.cc:237] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } }
I20260620 08:04:12.232583 1370 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: TxnStatusTablet state changed. Reason: New leader 622d99cf77474717b99a87ac39225e94. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.232676 1370 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:12.232897 1402 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:04:12.233028 1402 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:04:12.232954 853 catalog_manager.cc:5495] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 622d99cf77474717b99a87ac39225e94 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:12.251034 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.248178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40239 (local address 127.0.0.1:42642)
0620 08:04:12.248264 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:12.248278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:12.248334 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:12.248493 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:12.248494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:12.248495 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:12.248541 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.248543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.249425 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.249425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.250116 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.250118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.250592 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.250593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.250683 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.250701 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.250887 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.250894 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:04:12.250929 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 08:04:12.250971 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:04:12.251041 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.248265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.31.115.193:40239)
0620 08:04:12.248341 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.248344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.248355 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:12.248416 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.248417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.248429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.248481 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:12.248588 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.249329 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.250200 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.250508 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.250599 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.250786 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.250824 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.250877 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.250932 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.250955 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:04:12.252266 1291 raft_consensus.cc:1261] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Refusing update from remote peer 622d99cf77474717b99a87ac39225e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:12.252324 1151 raft_consensus.cc:1261] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Refusing update from remote peer 622d99cf77474717b99a87ac39225e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:12.252542 1380 consensus_queue.cc:1035] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:12.252712 1370 consensus_queue.cc:1035] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:12.253397 1364 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: TxnStatusTablet state changed. Reason: New leader 622d99cf77474717b99a87ac39225e94. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.253485 1364 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.253535 1365 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: TxnStatusTablet state changed. Reason: New leader 622d99cf77474717b99a87ac39225e94. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.253650 1365 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.254384 1370 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.254490 1370 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:12.254405 1380 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.254519 1380 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 622d99cf77474717b99a87ac39225e94: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:12.254734 1364 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.254812 1364 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P 628aaa2ba0a64061aadf2d209b22af05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.255038 1365 tablet_replica.cc:439] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "622d99cf77474717b99a87ac39225e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 } } peers { permanent_uuid: "628aaa2ba0a64061aadf2d209b22af05" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34581 } } peers { permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 } } }
I20260620 08:04:12.255127 1365 tablet_replica.cc:442] T c050b661030f4deea913abec8f90d5f2 P a874c913d59d41f3961e4a81b81d68bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:12.260727 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.258289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.31.115.194:34581)
0620 08:04:12.258369 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.258373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.258386 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:04:12.258436 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.258437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.258459 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.258548 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:12.258721 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.259194 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.259918 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.260127 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.260202 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.260344 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.260460 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.260545 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.260586 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.260605 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:04:12.268234 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.263435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:50669 (local address 127.31.115.193:40239)
0620 08:04:12.263504 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.263506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.263652 (+ 146us) server_negotiation.cc:407] Connection header received
0620 08:04:12.263688 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.263689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.263701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.263761 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:12.263880 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.264352 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.267488 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.267737 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.267816 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.268001 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.268030 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.268072 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.268149 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.268168 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:04:12.268287 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.263367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40239 (local address 127.31.115.194:50669)
0620 08:04:12.263544 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:12.263557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:12.263639 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:12.263763 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:12.263764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:12.263765 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:12.263823 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.263825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.264430 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.264431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.267378 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.267382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.267822 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.267824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.267894 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.267914 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.268092 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.268094 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:12.268142 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:04:12.268185 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260620 08:04:12.270877 1011 raft_consensus.cc:1261] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Refusing update from remote peer 628aaa2ba0a64061aadf2d209b22af05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:12.271168 1291 raft_consensus.cc:1261] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Refusing update from remote peer 628aaa2ba0a64061aadf2d209b22af05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:12.271220 1364 consensus_queue.cc:1035] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622d99cf77474717b99a87ac39225e94" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:12.271361 1390 consensus_queue.cc:1035] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a874c913d59d41f3961e4a81b81d68bc" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:12.280823 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.278101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:35733 (local address 127.31.115.194:34581)
0620 08:04:12.278148 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.278150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.278259 (+ 109us) server_negotiation.cc:407] Connection header received
0620 08:04:12.278328 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.278329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.278338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.278385 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:12.278498 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.279006 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.280110 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.280339 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.280392 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.280582 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.280609 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.280647 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.280735 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.280755 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:12.280869 1414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.278007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34581 (local address 127.31.115.193:35733)
0620 08:04:12.278192 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:12.278204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:12.278269 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:12.278388 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:12.278389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:12.278391 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:12.278443 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.278444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.279083 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.279084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.279999 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:12.280002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.280412 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.280413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:12.280475 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.280495 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:12.280667 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:12.280669 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:12.280733 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:04:12.280779 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260620 08:04:12.290719 853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:12.290858 853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:12.291549 853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b216ec759e97430080661291534bb2d6: Sending DeleteTablet for 3 replicas of tablet 8494f6d4c95449708a156b48a86f938d
I20260620 08:04:12.291865 1131 tablet_service.cc:1514] Processing DeleteTablet for tablet 8494f6d4c95449708a156b48a86f938d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:12 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:51956
I20260620 08:04:12.291965 1385 tablet_replica.cc:330] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: stopping tablet replica
I20260620 08:04:12.292011 1385 raft_consensus.cc:2229] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:12.292104 1385 raft_consensus.cc:2258] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:12.292943 1385 ts_tablet_manager.cc:1905] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:12.293828 991 tablet_service.cc:1514] Processing DeleteTablet for tablet 8494f6d4c95449708a156b48a86f938d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:12 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:42636
I20260620 08:04:12.293830 1271 tablet_service.cc:1514] Processing DeleteTablet for tablet 8494f6d4c95449708a156b48a86f938d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:12 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:35284
I20260620 08:04:12.293952 1383 tablet_replica.cc:330] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: stopping tablet replica
I20260620 08:04:12.294018 1383 raft_consensus.cc:2229] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:12.294067 1383 raft_consensus.cc:2258] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:12.294147 1384 tablet_replica.cc:330] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: stopping tablet replica
I20260620 08:04:12.294211 1384 raft_consensus.cc:2229] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:12.294263 1384 raft_consensus.cc:2258] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:12.294418 1383 ts_tablet_manager.cc:1905] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:12.294785 1385 ts_tablet_manager.cc:1918] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:12.294855 1385 log.cc:1196] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/8494f6d4c95449708a156b48a86f938d
I20260620 08:04:12.295080 1385 ts_tablet_manager.cc:1939] T 8494f6d4c95449708a156b48a86f938d P 628aaa2ba0a64061aadf2d209b22af05: Deleting consensus metadata
I20260620 08:04:12.295605 838 catalog_manager.cc:4841] TS 628aaa2ba0a64061aadf2d209b22af05 (127.31.115.194:34581): tablet 8494f6d4c95449708a156b48a86f938d (table test-table [id=ebbd1e7c2b844e2aaf0338e4661249fa]) successfully deleted
I20260620 08:04:12.296052 1383 ts_tablet_manager.cc:1918] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:12.296103 1383 log.cc:1196] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/8494f6d4c95449708a156b48a86f938d
I20260620 08:04:12.296355 1383 ts_tablet_manager.cc:1939] T 8494f6d4c95449708a156b48a86f938d P 622d99cf77474717b99a87ac39225e94: Deleting consensus metadata
I20260620 08:04:12.296772 1384 ts_tablet_manager.cc:1905] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:12.298563 1384 ts_tablet_manager.cc:1918] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:12.298627 1384 log.cc:1196] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/8494f6d4c95449708a156b48a86f938d
I20260620 08:04:12.298899 1384 ts_tablet_manager.cc:1939] T 8494f6d4c95449708a156b48a86f938d P a874c913d59d41f3961e4a81b81d68bc: Deleting consensus metadata
I20260620 08:04:12.299350 838 catalog_manager.cc:4841] TS a874c913d59d41f3961e4a81b81d68bc (127.31.115.195:43787): tablet 8494f6d4c95449708a156b48a86f938d (table test-table [id=ebbd1e7c2b844e2aaf0338e4661249fa]) successfully deleted
I20260620 08:04:12.299746 840 catalog_manager.cc:4841] TS 622d99cf77474717b99a87ac39225e94 (127.31.115.193:40239): tablet 8494f6d4c95449708a156b48a86f938d (table test-table [id=ebbd1e7c2b844e2aaf0338e4661249fa]) successfully deleted
Jun 20 08:04:12 dist-test-slave-c4w8 krb5kdc[809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:12.304737 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.292946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.31.115.193:40239)
0620 08:04:12.293707 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.293710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.294251 (+ 541us) server_negotiation.cc:407] Connection header received
0620 08:04:12.294423 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.294424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.294434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.294465 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:12.295663 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.296166 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.297274 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.297573 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.297645 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.301512 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:12.301525 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:12.301526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:12.301534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:12.303440 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:12.303853 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:12.303857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:12.303857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:12.303928 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:12.304136 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:12.304137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:12.304137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:12.304310 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:12.304496 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.304545 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.304629 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733}
W20260620 08:04:12.305115 1051 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42644
I20260620 08:04:12.316200 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:12.306523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.31.115.254:35567)
0620 08:04:12.306602 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:12.306604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:12.309856 (+ 3252us) server_negotiation.cc:407] Connection header received
0620 08:04:12.309941 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:12.309942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:12.309955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:12.310029 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:12.310592 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.311131 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.311867 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:12.312085 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:12.312169 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:12.313436 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:12.313438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:12.313438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:12.313447 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:12.314995 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:12.315379 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:12.315381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:12.315381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:12.315446 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:12.315674 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:12.315675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:12.315676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:12.315801 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:12.315913 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:12.316009 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 08:04:12.316093 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 08:04:12.316560 853 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59692
I20260620 08:04:12.317140 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 926
I20260620 08:04:12.324777 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 1066
I20260620 08:04:12.330180 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 1206
I20260620 08:04:12.334889 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 822
2026-06-20T08:04:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2419 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:12 dist-test-slave-c4w8 krb5kdc[1423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:12 dist-test-slave-c4w8 krb5kdc[1423](info): set up 2 sockets
Jun 20 08:04:12 dist-test-slave-c4w8 krb5kdc[1423](info): commencing operation
krb5kdc: starting...
W20260620 08:04:13.393931 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:13 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:13.520761 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:44239
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:13.607766 1436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:13.608028 1436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:13.608069 1436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:13.609308 1436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:13.609388 1436 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:04:13.609411 1436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:13.609432 1436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:13.609464 1436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:13.609491 1436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:13.611141 1436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:44239
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:13.611462 1436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:13.611721 1436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:13.617215 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:13.617434 1436 server_base.cc:1034] running on GCE node
W20260620 08:04:13.617204 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:13.617231 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:13.617909 1436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:13.618256 1436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:13.619463 1436 hybrid_clock.cc:648] HybridClock initialized: now 1781942653619451 us; error 49 us; skew 500 ppm
Jun 20 08:04:13 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:13.622352 1436 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:13.622874 1436 webserver.cc:460] Webserver started at http://127.31.115.254:41083/ using document root <none> and password file <none>
I20260620 08:04:13.623116 1436 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:13.623162 1436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:13.623278 1436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:13.624433 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "571b136b05284297b5585f6dc90978fd"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "52e5dfb72d8e7454367d1354ff9f21fb"
server_key_iv: "8bd58faa2031785a651c157af7c74bd3"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.624769 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "571b136b05284297b5585f6dc90978fd"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "52e5dfb72d8e7454367d1354ff9f21fb"
server_key_iv: "8bd58faa2031785a651c157af7c74bd3"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.626422 1436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:13.627290 1451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:13.627521 1436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:13.627617 1436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "571b136b05284297b5585f6dc90978fd"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "52e5dfb72d8e7454367d1354ff9f21fb"
server_key_iv: "8bd58faa2031785a651c157af7c74bd3"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.627681 1436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:13.650557 1436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:13.653025 1436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:13.653141 1436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:13.658196 1436 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:44239
I20260620 08:04:13.658380 1513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:44239 every 8 connection(s)
I20260620 08:04:13.658581 1436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:13.659380 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:13.662030 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Bootstrap starting.
I20260620 08:04:13.662701 1514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:13.663012 1514 log.cc:824] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:13.663810 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: No bootstrap required, opened a new log
I20260620 08:04:13.666155 1514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b136b05284297b5585f6dc90978fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44239 } }
I20260620 08:04:13.666324 1514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:13.666373 1514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b136b05284297b5585f6dc90978fd, State: Initialized, Role: FOLLOWER
I20260620 08:04:13.666548 1514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b136b05284297b5585f6dc90978fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44239 } }
I20260620 08:04:13.666617 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 1436
I20260620 08:04:13.666661 1514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:13.666706 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:13.666705 1514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:13.666762 1514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:13.666891 32207 external_mini_cluster.cc:1428] Setting key 78cff59d07a45e7e1c57397ed5b50bd1
I20260620 08:04:13.667430 1514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b136b05284297b5585f6dc90978fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44239 } }
I20260620 08:04:13.667550 1514 leader_election.cc:302] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571b136b05284297b5585f6dc90978fd; no voters:
I20260620 08:04:13.667733 1514 leader_election.cc:288] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:13.667804 1519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:13.667955 1519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [term 1 LEADER]: Becoming Leader. State: Replica: 571b136b05284297b5585f6dc90978fd, State: Running, Role: LEADER
I20260620 08:04:13.667989 1514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:13.668069 1519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b136b05284297b5585f6dc90978fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44239 } }
I20260620 08:04:13.670055 1521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571b136b05284297b5585f6dc90978fd. Latest consensus state: current_term: 1 leader_uuid: "571b136b05284297b5585f6dc90978fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b136b05284297b5585f6dc90978fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44239 } } }
I20260620 08:04:13.669809 1520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "571b136b05284297b5585f6dc90978fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b136b05284297b5585f6dc90978fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44239 } } }
I20260620 08:04:13.670235 1520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:13.670378 1521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd [sys.catalog]: This master's current role is: LEADER
W20260620 08:04:13.670475 1533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:04:13.670521 1533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:04:13.670637 1536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:13.670871 1536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:13.672581 1536 catalog_manager.cc:1347] Generated new cluster ID: 964a76e0a55e434393d2b83db19bb887
I20260620 08:04:13.672652 1536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:13 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:13.680079 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:13.667421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.31.115.254:44239)
0620 08:04:13.667630 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:13.667639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:13.667678 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:04:13.667806 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:13.667815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:13.667877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:13.668839 (+ 962us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:13.668957 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.670593 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.671688 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.672161 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.672307 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:13.675247 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:13.675252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:13.675253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:13.675266 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:13.678004 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:13.678515 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:13.678519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:13.678520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:13.678672 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:13.678913 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:13.678915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:13.678915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:13.679296 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:13.679469 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:13.679549 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 08:04:13.679834 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260620 08:04:13.692409 1536 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:13.693014 1536 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:13.697750 1536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Generated new TSK 0
I20260620 08:04:13.697978 1536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:13.729418 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44239
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:13.814883 1540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:13.815084 1540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:13.815111 1540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:13.816293 1540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:13.816337 1540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:13.816358 1540 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:04:13.816370 1540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:13.816432 1540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:13.817957 1540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:13.818218 1540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:13.818462 1540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:04:13.824414 1540 server_base.cc:1034] running on GCE node
W20260620 08:04:13.824348 1545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:13.824359 1546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:13.824328 1548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:13.824822 1540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:13.825088 1540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:13.826272 1540 hybrid_clock.cc:648] HybridClock initialized: now 1781942653826249 us; error 45 us; skew 500 ppm
Jun 20 08:04:13 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:13.829068 1540 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:13.829584 1540 webserver.cc:460] Webserver started at http://127.31.115.193:43383/ using document root <none> and password file <none>
I20260620 08:04:13.829960 1540 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:13.830013 1540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:13.830158 1540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:13.831538 1540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "4a4a3ea65888419281984db1f94bf67c"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "ab9f6bc56823f8ab9f4224642acac78c"
server_key_iv: "17a9d5280cdeb8609ac9835e4f9d8a04"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.832022 1540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "4a4a3ea65888419281984db1f94bf67c"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "ab9f6bc56823f8ab9f4224642acac78c"
server_key_iv: "17a9d5280cdeb8609ac9835e4f9d8a04"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.833639 1540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 08:04:13.834667 1555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:13.834949 1540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:13.835048 1540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "4a4a3ea65888419281984db1f94bf67c"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "ab9f6bc56823f8ab9f4224642acac78c"
server_key_iv: "17a9d5280cdeb8609ac9835e4f9d8a04"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.835134 1540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:13.849375 1540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:13.851913 1540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:13.852105 1540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:13.852774 1540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:13.852815 1540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:13.852839 1540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:13.852851 1540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:04:13 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:13.860383 1540 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:33133
I20260620 08:04:13.860503 1670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:33133 every 8 connection(s)
I20260620 08:04:13.860816 1540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:13.863571 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:13.853182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:37029 (local address 127.31.115.254:44239)
0620 08:04:13.853265 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:13.853268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:13.853550 (+ 282us) server_negotiation.cc:407] Connection header received
0620 08:04:13.854344 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:13.854345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:13.854363 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:13.854429 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:13.855414 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.855937 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.857689 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.857905 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.857994 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:13.860268 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:13.860270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:13.860270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:13.860279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:13.862001 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:13.862490 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:13.862492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:13.862493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:13.862554 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:13.862856 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:13.862857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:13.862857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:13.862986 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:13.863061 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:13.863386 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 08:04:13.863480 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:04:13.863785 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:13.853171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.31.115.193:37029)
0620 08:04:13.853392 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:13.853449 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:13.854213 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:13.854518 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:13.854524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:13.854963 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:13.855300 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:13.855308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.856042 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.856043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:13.857549 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:13.857553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.858026 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.858028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:13.858166 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:13.858722 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:13.858738 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:13.860150 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:13.862118 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:13.862125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:13.862127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:13.862389 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:13.862656 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:13.862657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:13.862658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:13.862776 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:13.863063 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:13.863071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:13.863228 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 08:04:13.863350 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 08:04:13.863522 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260620 08:04:13.864454 1671 heartbeater.cc:347] Connected to a master server at 127.31.115.254:44239
I20260620 08:04:13.864540 1671 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:13.864850 1671 heartbeater.cc:511] Master 127.31.115.254:44239 requested a full tablet report, sending...
I20260620 08:04:13.865223 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 1540
I20260620 08:04:13.865336 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:13.865505 32207 external_mini_cluster.cc:1428] Setting key 81b541ef4209d281b5680e4e00e0eda6
I20260620 08:04:13.865525 1467 ts_manager.cc:194] Registered new tserver with Master: 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133)
I20260620 08:04:13.866535 1467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:37029
I20260620 08:04:13.872431 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:13.865729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:33333 (local address 127.31.115.254:44239)
0620 08:04:13.865771 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:13.865772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:13.865958 (+ 186us) server_negotiation.cc:407] Connection header received
0620 08:04:13.866007 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:13.866008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:13.866019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:13.866078 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:13.866532 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.867285 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.868382 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.868573 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.868762 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:13.869864 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:13.869866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:13.869866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:13.869873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:13.871236 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:13.871602 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:13.871604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:13.871604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:13.871658 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:13.871867 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:13.871868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:13.871869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:13.871989 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:13.872081 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:13.872241 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:04:13.872331 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:13.872437 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:13.865726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.31.115.193:33333)
0620 08:04:13.865873 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:13.865885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:13.865952 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:13.866184 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:13.866185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:13.866378 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:13.866440 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:13.866442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.867394 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.867395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:13.868283 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:13.868287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:13.868784 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:13.868785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:13.868880 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:13.869357 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:13.869361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:13.869761 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:13.871335 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:13.871336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:13.871336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:13.871514 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:13.871733 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:13.871734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:13.871735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:13.871795 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:13.872072 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:13.872074 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:13.872126 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:04:13.872216 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:04:13.872331 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:13.893209 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44239
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:13.977942 1680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:13.978164 1680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:13.978192 1680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:13.979517 1680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:13.979569 1680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:13.979591 1680 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:04:13.979604 1680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:13.979622 1680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:13.981170 1680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:13.981448 1680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:13.981724 1680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:04:13.987169 1680 server_base.cc:1034] running on GCE node
W20260620 08:04:13.987103 1688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:13.987097 1686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:13.987103 1685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:13.987526 1680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:13.987841 1680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:13.989006 1680 hybrid_clock.cc:648] HybridClock initialized: now 1781942653988988 us; error 30 us; skew 500 ppm
Jun 20 08:04:13 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:13.991894 1680 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:13.992462 1680 webserver.cc:460] Webserver started at http://127.31.115.194:45493/ using document root <none> and password file <none>
I20260620 08:04:13.992738 1680 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:13.992786 1680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:13.992882 1680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:13.994055 1680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "4085f29540ea4be98ff5aa929a768d4b"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "a15321ef7b41aeb161fd2b2a2c770323"
server_key_iv: "ab07aa629071facdcd5d62b05560bc48"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.994426 1680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "4085f29540ea4be98ff5aa929a768d4b"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "a15321ef7b41aeb161fd2b2a2c770323"
server_key_iv: "ab07aa629071facdcd5d62b05560bc48"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.995810 1680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:13.996743 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:13.996987 1680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:13.997104 1680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "4085f29540ea4be98ff5aa929a768d4b"
format_stamp: "Formatted at 2026-06-20 08:04:13 on dist-test-slave-c4w8"
server_key: "a15321ef7b41aeb161fd2b2a2c770323"
server_key_iv: "ab07aa629071facdcd5d62b05560bc48"
server_key_version: "encryptionkey@0"
I20260620 08:04:13.997184 1680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:14.017573 1680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:14.020164 1680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:14.020349 1680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:14.021023 1680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:14.021095 1680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:14.021142 1680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:14.021169 1680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:04:14 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:14.029134 1680 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:45385
I20260620 08:04:14.029253 1810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:45385 every 8 connection(s)
I20260620 08:04:14.029577 1680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:14.031795 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.021472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:44087 (local address 127.31.115.254:44239)
0620 08:04:14.021545 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.021548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.021879 (+ 331us) server_negotiation.cc:407] Connection header received
0620 08:04:14.022405 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.022407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.022425 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.022509 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:14.023209 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.024061 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.025131 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.025351 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.025431 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.028237 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:14.028239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:14.028239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:14.028249 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:14.030215 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.030709 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.030711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.030711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.030784 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.031050 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.031051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.031051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.031185 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:14.031259 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.031596 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.031701 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:04:14.031985 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.021447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.31.115.194:44087)
0620 08:04:14.021627 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.021670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.022302 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.022610 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.022615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.022878 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:14.023084 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.023089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.024188 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.024189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.025024 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.025027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.025454 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.025455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.025582 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.026223 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:14.026235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:14.028113 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:14.030321 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.030328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.030330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.030618 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.030854 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.030855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.030856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.030972 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.031260 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:14.031268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:14.031442 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.031572 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.031747 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260620 08:04:14.032509 1811 heartbeater.cc:347] Connected to a master server at 127.31.115.254:44239
I20260620 08:04:14.032612 1811 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:14.032819 1811 heartbeater.cc:511] Master 127.31.115.254:44239 requested a full tablet report, sending...
I20260620 08:04:14.033272 1468 ts_manager.cc:194] Registered new tserver with Master: 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385)
I20260620 08:04:14.033814 1468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:44087
I20260620 08:04:14.038836 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 1680
I20260620 08:04:14.038939 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:14.039198 32207 external_mini_cluster.cc:1428] Setting key 8b790bc5516b849b4bd70100065d2909
I20260620 08:04:14.041848 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.033648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.31.115.194:44473)
0620 08:04:14.033773 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.033783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.033843 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.034013 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.034014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.034224 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:14.034286 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.034288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.035094 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.035095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.035612 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.035614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.035988 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.035990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.036050 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.036568 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:14.036574 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:14.036893 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:14.038931 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.038932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.038935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.039244 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.039476 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.039477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.039477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.039581 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.041392 (+ 1811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:14.041392 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:14.041447 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.041543 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.041771 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260620 08:04:14.041949 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.033707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:44473 (local address 127.31.115.254:44239)
0620 08:04:14.033749 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.033752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.033825 (+ 73us) server_negotiation.cc:407] Connection header received
0620 08:04:14.033923 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.033924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.033935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.033987 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:14.034368 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.034997 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.035721 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.035901 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.035957 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.036989 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:14.036990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:14.036991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:14.036997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:14.038795 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.039331 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.039333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.039334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.039397 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.039706 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.039707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.039707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.039994 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:14.040099 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.041722 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.041826 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:14.065337 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44239
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:14.147686 1820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:14.147908 1820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:14.147935 1820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:14.149122 1820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:14.149166 1820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:14.149188 1820 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:04:14.149201 1820 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:14.149219 1820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:14.150789 1820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:14.151060 1820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:14.151301 1820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:14.156600 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:14.156630 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:14.156600 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:14.156773 1820 server_base.cc:1034] running on GCE node
I20260620 08:04:14.156975 1820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:14.157243 1820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:14.158385 1820 hybrid_clock.cc:648] HybridClock initialized: now 1781942654158372 us; error 35 us; skew 500 ppm
Jun 20 08:04:14 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:14.160871 1820 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:14.161338 1820 webserver.cc:460] Webserver started at http://127.31.115.195:33345/ using document root <none> and password file <none>
I20260620 08:04:14.161580 1820 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:14.161689 1820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:14.161842 1820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:14.162904 1820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "8011097ac3714be19bc4ed5784274169"
format_stamp: "Formatted at 2026-06-20 08:04:14 on dist-test-slave-c4w8"
server_key: "3b7ebaee75d8d0165c926d82808396a1"
server_key_iv: "8c3be3c7252e7b0ca42395b9f9b1b9ed"
server_key_version: "encryptionkey@0"
I20260620 08:04:14.163271 1820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "8011097ac3714be19bc4ed5784274169"
format_stamp: "Formatted at 2026-06-20 08:04:14 on dist-test-slave-c4w8"
server_key: "3b7ebaee75d8d0165c926d82808396a1"
server_key_iv: "8c3be3c7252e7b0ca42395b9f9b1b9ed"
server_key_version: "encryptionkey@0"
I20260620 08:04:14.164780 1820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:14.165697 1835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:14.165887 1820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:14.165992 1820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "8011097ac3714be19bc4ed5784274169"
format_stamp: "Formatted at 2026-06-20 08:04:14 on dist-test-slave-c4w8"
server_key: "3b7ebaee75d8d0165c926d82808396a1"
server_key_iv: "8c3be3c7252e7b0ca42395b9f9b1b9ed"
server_key_version: "encryptionkey@0"
I20260620 08:04:14.166075 1820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:14.189879 1820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:14.191926 1820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:14.192116 1820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:14.192821 1820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:14.192884 1820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:14.192938 1820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:14.192970 1820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:04:14 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942654, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:14.201077 1820 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:40051
I20260620 08:04:14.201442 1820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:14.201777 1950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:40051 every 8 connection(s)
I20260620 08:04:14.203693 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.193280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:40601 (local address 127.31.115.254:44239)
0620 08:04:14.193344 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.193351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.193556 (+ 205us) server_negotiation.cc:407] Connection header received
0620 08:04:14.194133 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.194135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.194148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.194213 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:14.194923 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.195421 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.196791 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.197013 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.197123 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.200442 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:14.200444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:14.200444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:14.200454 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:14.202305 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.202719 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.202720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.202721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.202783 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.203031 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.203032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.203032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.203158 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:14.203219 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.203551 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.203626 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:04:14.203871 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.193275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.31.115.195:40601)
0620 08:04:14.193429 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.193464 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.194019 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.194289 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.194292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.194593 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:14.194823 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.194827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.195517 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.195518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.196665 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.196674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.197130 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.197131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.197381 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.197972 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:14.197982 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:14.200323 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:14.202383 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.202387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.202389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.202641 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.202849 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.202850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.202850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.202951 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.203254 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:14.203264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:14.203394 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.203493 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.203637 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260620 08:04:14.204408 1951 heartbeater.cc:347] Connected to a master server at 127.31.115.254:44239
I20260620 08:04:14.204492 1951 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:14.204747 1951 heartbeater.cc:511] Master 127.31.115.254:44239 requested a full tablet report, sending...
I20260620 08:04:14.205154 1467 ts_manager.cc:194] Registered new tserver with Master: 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:14.205600 1467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:40601
I20260620 08:04:14.211139 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 1820
I20260620 08:04:14.211267 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:14.211467 32207 external_mini_cluster.cc:1428] Setting key 115490c45ff2fa3c76b847a8aaa9bc8b
I20260620 08:04:14.211493 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.205752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:55001 (local address 127.31.115.254:44239)
0620 08:04:14.205790 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.205792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.205920 (+ 128us) server_negotiation.cc:407] Connection header received
0620 08:04:14.205969 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.205969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.205980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.206031 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:14.206423 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.206918 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.207652 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.207840 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.207896 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.208902 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:14.208904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:14.208904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:14.208910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:14.210260 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.210637 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.210639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.210639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.210719 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.210903 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.210904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.210904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.211024 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:14.211117 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.211312 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.211395 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:04:14.211508 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.205600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.31.115.195:55001)
0620 08:04:14.205843 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.205855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.205919 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.206042 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.206043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.206254 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:14.206337 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.206339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.207032 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.207033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.207570 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.207572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.207912 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.207913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.207965 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.208457 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:14.208463 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:14.208821 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:14.210354 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.210355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.210355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.210547 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.210787 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.210788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.210788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.210846 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.211120 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:14.211122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:14.211209 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.211286 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.211397 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260620 08:04:14.212409 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:04:14 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:04:14 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:14.236135 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.228404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.31.115.254:44239)
0620 08:04:14.228465 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.228467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.228476 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:14.228522 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.228523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.228533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.228582 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:14.229149 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.229819 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.230616 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.230819 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.230876 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.233434 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:14.233436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:14.233437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:14.233452 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:14.234953 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.235336 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.235337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.235337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.235397 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.235595 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.235596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.235597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.235724 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:14.235828 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.235939 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.236025 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:04:14.238039 1467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:14.239771 1467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:14.246886 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.242476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:40051 (local address 127.0.0.1:59574)
0620 08:04:14.242836 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.242847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.242900 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.243420 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.243421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.243423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.243479 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.243481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.244781 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.244782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.245804 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.245807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.246565 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.246647 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.246660 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.246734 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.246792 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260620 08:04:14.246891 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.242861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.31.115.195:40051)
0620 08:04:14.243237 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.243255 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.243265 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:14.243327 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.243329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.243344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.243413 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.243561 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.244675 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.245877 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.246468 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246565 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.246626 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.246756 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.246796 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":80,"threads_started":1}
I20260620 08:04:14.247272 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.242846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.31.115.193:33133)
0620 08:04:14.243014 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.243019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.243105 (+ 86us) server_negotiation.cc:407] Connection header received
0620 08:04:14.243182 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.243184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.243201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.243295 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.243564 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.244400 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.245811 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.246561 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246636 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.246688 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.247113 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.247156 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260620 08:04:14.247434 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.242670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.31.115.194:45385)
0620 08:04:14.243535 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.243539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.243549 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:14.243585 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.243587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.243603 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.243679 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.243793 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.244819 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246083 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.246788 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246858 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.246905 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.247295 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.247337 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":79,"threads_started":1}
I20260620 08:04:14.247545 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.242475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.0.0.1:34494)
0620 08:04:14.242632 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.242666 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.242759 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.243693 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.243693 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.243695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.243730 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.243731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.244914 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.244915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.245998 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.245999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.246937 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.247197 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.247201 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.247275 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.247320 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"spinlock_wait_cycles":768,"thread_start_us":93,"threads_started":1}
I20260620 08:04:14.247807 1971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.242684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33133 (local address 127.0.0.1:33116)
0620 08:04:14.243028 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.243057 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.243112 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.243390 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.243394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.243396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.243465 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.243467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.244498 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.244499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.245725 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.245727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.246738 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.246739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.246813 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.246818 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.246864 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.246905 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":104,"threads_started":1}
I20260620 08:04:14.247650 1885 tablet_service.cc:1467] Processing CreateTablet for tablet 98b1ef9b4a7f4115b005786511a1a523 (DEFAULT_TABLE table=test-table [id=086b693e573f4ef9b352a882593767de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:14.248044 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b1ef9b4a7f4115b005786511a1a523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.248310 1745 tablet_service.cc:1467] Processing CreateTablet for tablet 98b1ef9b4a7f4115b005786511a1a523 (DEFAULT_TABLE table=test-table [id=086b693e573f4ef9b352a882593767de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:14.248617 1745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b1ef9b4a7f4115b005786511a1a523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.250300 1605 tablet_service.cc:1467] Processing CreateTablet for tablet 98b1ef9b4a7f4115b005786511a1a523 (DEFAULT_TABLE table=test-table [id=086b693e573f4ef9b352a882593767de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:14.250629 1974 tablet_bootstrap.cc:492] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Bootstrap starting.
I20260620 08:04:14.250804 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b1ef9b4a7f4115b005786511a1a523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.251458 1974 tablet_bootstrap.cc:654] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.251780 1974 log.cc:824] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:14.252554 1974 tablet_bootstrap.cc:492] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: No bootstrap required, opened a new log
I20260620 08:04:14.252624 1974 ts_tablet_manager.cc:1397] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:14.253804 1976 tablet_bootstrap.cc:492] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Bootstrap starting.
I20260620 08:04:14.254247 1977 tablet_bootstrap.cc:492] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Bootstrap starting.
I20260620 08:04:14.254331 1974 raft_consensus.cc:348] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.254475 1976 tablet_bootstrap.cc:654] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.254513 1974 raft_consensus.cc:374] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.254549 1974 raft_consensus.cc:729] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8011097ac3714be19bc4ed5784274169, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.254727 1974 consensus_queue.cc:260] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.254796 1976 log.cc:824] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:14.254880 1977 tablet_bootstrap.cc:654] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.255008 1974 ts_tablet_manager.cc:1428] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 08:04:14.255081 1951 heartbeater.cc:503] Master 127.31.115.254:44239 was elected leader, sending a full tablet report...
I20260620 08:04:14.255231 1977 log.cc:824] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:14.256042 1976 tablet_bootstrap.cc:492] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: No bootstrap required, opened a new log
I20260620 08:04:14.256122 1976 ts_tablet_manager.cc:1397] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:14.256332 1977 tablet_bootstrap.cc:492] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: No bootstrap required, opened a new log
I20260620 08:04:14.256412 1977 ts_tablet_manager.cc:1397] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:14.257666 1976 raft_consensus.cc:348] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.257831 1976 raft_consensus.cc:374] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.257876 1976 raft_consensus.cc:729] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.257887 1977 raft_consensus.cc:348] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.257995 1976 consensus_queue.cc:260] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.258037 1977 raft_consensus.cc:374] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.258095 1977 raft_consensus.cc:729] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a3ea65888419281984db1f94bf67c, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.258225 1977 consensus_queue.cc:260] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.258270 1976 ts_tablet_manager.cc:1428] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:14.258334 1811 heartbeater.cc:503] Master 127.31.115.254:44239 was elected leader, sending a full tablet report...
I20260620 08:04:14.258489 1977 ts_tablet_manager.cc:1428] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:14.258877 1671 heartbeater.cc:503] Master 127.31.115.254:44239 was elected leader, sending a full tablet report...
W20260620 08:04:14.280369 1812 tablet.cc:2369] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:14.296595 1982 raft_consensus.cc:482] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:14.296767 1982 raft_consensus.cc:504] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.297261 1982 leader_election.cc:288] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051), 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385)
I20260620 08:04:14.300350 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.297460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:40051 (local address 127.31.115.193:43165)
0620 08:04:14.297531 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.297544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.297594 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.297828 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.297830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.297832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.297912 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.297914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.298617 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.298618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.299528 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.299531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.300092 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.300093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.300176 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.300180 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.300235 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.300270 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 08:04:14.300400 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.297562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:43165 (local address 127.31.115.195:40051)
0620 08:04:14.297641 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.297644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.297655 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:14.297739 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.297740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.297751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.297811 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.298003 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.298536 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.299642 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.300010 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.300068 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.300076 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.300250 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.300293 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:14.300493 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.297497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.31.115.193:59815)
0620 08:04:14.297705 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.297717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.297785 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.297939 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.297940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.297941 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.298052 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.298054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.298750 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.298751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.299534 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.299536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.300196 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.300198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.300293 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.300298 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.300356 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.300393 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260620 08:04:14.300526 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.297625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:59815 (local address 127.31.115.194:45385)
0620 08:04:14.297670 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.297672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.297825 (+ 153us) server_negotiation.cc:407] Connection header received
0620 08:04:14.297853 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.297854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.297863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.297917 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.298125 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.298656 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.299640 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.300106 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.300177 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.300185 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.300377 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.300441 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:04:14.300747 1902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b1ef9b4a7f4115b005786511a1a523" candidate_uuid: "4a4a3ea65888419281984db1f94bf67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169" is_pre_election: true
I20260620 08:04:14.300818 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b1ef9b4a7f4115b005786511a1a523" candidate_uuid: "4a4a3ea65888419281984db1f94bf67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4085f29540ea4be98ff5aa929a768d4b" is_pre_election: true
I20260620 08:04:14.300908 1902 raft_consensus.cc:2454] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4a3ea65888419281984db1f94bf67c in term 0.
I20260620 08:04:14.300994 1765 raft_consensus.cc:2454] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4a3ea65888419281984db1f94bf67c in term 0.
I20260620 08:04:14.301201 1556 leader_election.cc:302] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4a3ea65888419281984db1f94bf67c, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:14.301342 1982 raft_consensus.cc:2781] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:14.301399 1982 raft_consensus.cc:482] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:14.301424 1982 raft_consensus.cc:3037] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.302181 1982 raft_consensus.cc:504] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.302357 1982 leader_election.cc:288] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [CANDIDATE]: Term 1 election: Requested vote from peers 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051), 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385)
I20260620 08:04:14.302497 1902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b1ef9b4a7f4115b005786511a1a523" candidate_uuid: "4a4a3ea65888419281984db1f94bf67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169"
I20260620 08:04:14.302551 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b1ef9b4a7f4115b005786511a1a523" candidate_uuid: "4a4a3ea65888419281984db1f94bf67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4085f29540ea4be98ff5aa929a768d4b"
I20260620 08:04:14.302589 1902 raft_consensus.cc:3037] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.302645 1765 raft_consensus.cc:3037] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.303346 1902 raft_consensus.cc:2454] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4a3ea65888419281984db1f94bf67c in term 1.
I20260620 08:04:14.303375 1765 raft_consensus.cc:2454] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4a3ea65888419281984db1f94bf67c in term 1.
I20260620 08:04:14.303530 1556 leader_election.cc:302] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4a3ea65888419281984db1f94bf67c, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:14.303654 1982 raft_consensus.cc:2781] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:14.303830 1982 raft_consensus.cc:686] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 1 LEADER]: Becoming Leader. State: Replica: 4a4a3ea65888419281984db1f94bf67c, State: Running, Role: LEADER
I20260620 08:04:14.303928 1982 consensus_queue.cc:237] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } }
I20260620 08:04:14.304839 1468 catalog_manager.cc:5495] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a4a3ea65888419281984db1f94bf67c (127.31.115.193). New cstate: current_term: 1 leader_uuid: "4a4a3ea65888419281984db1f94bf67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } health_report { overall_health: HEALTHY } } }
Jun 20 08:04:14 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942653, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:14.354445 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.346550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44239 (local address 127.0.0.1:43198)
0620 08:04:14.346776 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.346787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.346863 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.347045 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.347046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.347402 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:14.347456 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.347458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.348206 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.348207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.348868 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.348871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.349224 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.349224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.349271 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.349978 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:14.349983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:14.351580 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:14.353217 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.353225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.353228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.353437 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.353758 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:14.353759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:14.353759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:14.353839 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:14.354141 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:14.354148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:14.354204 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.354261 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.354367 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260620 08:04:14.354573 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.346693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.31.115.254:44239)
0620 08:04:14.346791 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.346793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.346888 (+ 95us) server_negotiation.cc:407] Connection header received
0620 08:04:14.346920 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.346921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.346933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.346991 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:14.347559 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.348103 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.348965 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.349148 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.349209 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.351668 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:14.351669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:14.351670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:14.351676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:14.353127 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.353539 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.353540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.353540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.353673 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:14.353910 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:14.353912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:14.353912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:14.354047 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:14.354262 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.354305 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.354426 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":62}
I20260620 08:04:14.357052 1468 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:43198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:04:14.357298 1468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:14.359191 1745 tablet_service.cc:1467] Processing CreateTablet for tablet 34c0d62c79304c5e84440eafb530f142 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ebef4c1c6d84000857a3260bd7c1772]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:14.359201 1605 tablet_service.cc:1467] Processing CreateTablet for tablet 34c0d62c79304c5e84440eafb530f142 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ebef4c1c6d84000857a3260bd7c1772]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:14.359368 1745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c0d62c79304c5e84440eafb530f142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.359375 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c0d62c79304c5e84440eafb530f142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.360562 1976 tablet_bootstrap.cc:492] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: Bootstrap starting.
I20260620 08:04:14.360730 1885 tablet_service.cc:1467] Processing CreateTablet for tablet 34c0d62c79304c5e84440eafb530f142 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1ebef4c1c6d84000857a3260bd7c1772]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:14.360920 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c0d62c79304c5e84440eafb530f142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.361294 1976 tablet_bootstrap.cc:654] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.361735 1977 tablet_bootstrap.cc:492] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: Bootstrap starting.
W20260620 08:04:14.361900 1672 tablet.cc:2369] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:14.362207 1977 tablet_bootstrap.cc:654] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.362373 1974 tablet_bootstrap.cc:492] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: Bootstrap starting.
I20260620 08:04:14.362785 1976 tablet_bootstrap.cc:492] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: No bootstrap required, opened a new log
I20260620 08:04:14.362854 1976 ts_tablet_manager.cc:1397] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:14.362875 1977 tablet_bootstrap.cc:492] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: No bootstrap required, opened a new log
I20260620 08:04:14.362914 1974 tablet_bootstrap.cc:654] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.362931 1977 ts_tablet_manager.cc:1397] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:14.363015 1976 raft_consensus.cc:348] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.363109 1976 raft_consensus.cc:374] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.363138 1976 raft_consensus.cc:729] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.363087 1977 raft_consensus.cc:348] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.363196 1977 raft_consensus.cc:374] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.363189 1976 consensus_queue.cc:260] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.363236 1977 raft_consensus.cc:729] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a3ea65888419281984db1f94bf67c, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.363304 1976 ts_tablet_manager.cc:1428] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:14.363301 1977 consensus_queue.cc:260] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.363391 1977 ts_tablet_manager.cc:1428] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:14.363373 1981 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.363430 1982 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.363458 1981 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:14.363488 1982 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:14.363780 1974 tablet_bootstrap.cc:492] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: No bootstrap required, opened a new log
I20260620 08:04:14.363828 1974 ts_tablet_manager.cc:1397] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:14.363976 1974 raft_consensus.cc:348] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.364049 1974 raft_consensus.cc:374] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.364118 1974 raft_consensus.cc:729] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8011097ac3714be19bc4ed5784274169, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.364171 1974 consensus_queue.cc:260] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.364261 1974 ts_tablet_manager.cc:1428] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:14.364324 1978 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.364400 1978 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 08:04:14.452315 1952 tablet.cc:2369] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:14.466566 1981 raft_consensus.cc:482] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:14.466725 1981 raft_consensus.cc:504] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.467165 1981 leader_election.cc:288] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:14.470721 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.467388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33133 (local address 127.31.115.194:38177)
0620 08:04:14.467445 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.467459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.467507 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.467830 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.467831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.467832 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.467922 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.467925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.468749 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.468750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.469784 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.469786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.470438 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.470439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.470541 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.470544 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.470599 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.470633 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 08:04:14.470772 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.467604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:38177 (local address 127.31.115.193:33133)
0620 08:04:14.467671 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.467673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.467682 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:14.467711 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.467711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.467721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.467774 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.468033 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.468660 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.469883 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.470339 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.470406 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.470414 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.470630 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.470691 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:14.470861 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.467675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:47379 (local address 127.31.115.195:40051)
0620 08:04:14.467747 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.467749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.467908 (+ 159us) server_negotiation.cc:407] Connection header received
0620 08:04:14.467946 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.467947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.467957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.468017 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:14.468252 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.468809 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.469956 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.470433 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.470505 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.470516 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.470744 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.470787 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:04:14.471091 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.467521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:40051 (local address 127.31.115.194:47379)
0620 08:04:14.467814 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.467824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.467901 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.468062 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.468062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.468064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:14.468138 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.468141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.468885 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.468887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.469847 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.469851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.470550 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.470551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.470644 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.470649 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.470719 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.470761 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260620 08:04:14.471086 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c0d62c79304c5e84440eafb530f142" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c" is_pre_election: true
I20260620 08:04:14.471225 1625 raft_consensus.cc:2454] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 0.
I20260620 08:04:14.471382 1902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c0d62c79304c5e84440eafb530f142" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169" is_pre_election: true
I20260620 08:04:14.471477 1902 raft_consensus.cc:2454] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 0.
I20260620 08:04:14.471439 1696 leader_election.cc:302] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 4a4a3ea65888419281984db1f94bf67c; no voters:
I20260620 08:04:14.471601 1981 raft_consensus.cc:2781] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:14.471654 1981 raft_consensus.cc:482] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:14.471678 1981 raft_consensus.cc:3037] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.497731 1981 raft_consensus.cc:504] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.497954 1981 leader_election.cc:288] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 election: Requested vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:14.498221 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c0d62c79304c5e84440eafb530f142" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c"
I20260620 08:04:14.498338 1625 raft_consensus.cc:3037] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.498289 1902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c0d62c79304c5e84440eafb530f142" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169"
I20260620 08:04:14.498399 1902 raft_consensus.cc:3037] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.499063 1902 raft_consensus.cc:2454] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 1.
I20260620 08:04:14.499315 1696 leader_election.cc:302] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:14.499351 1625 raft_consensus.cc:2454] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 1.
I20260620 08:04:14.499440 1981 raft_consensus.cc:2781] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:14.499570 1981 raft_consensus.cc:686] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [term 1 LEADER]: Becoming Leader. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Running, Role: LEADER
I20260620 08:04:14.499686 1981 consensus_queue.cc:237] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.500224 1996 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: TxnStatusTablet state changed. Reason: New leader 4085f29540ea4be98ff5aa929a768d4b. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.500339 1996 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:14.500531 1998 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:04:14.500669 1998 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:04:14.500627 1467 catalog_manager.cc:5495] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194). New cstate: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:14.514385 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.511413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.31.115.194:45385)
0620 08:04:14.511486 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.511488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.511496 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:14.511576 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.511577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.511593 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.511646 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:14.511795 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.512279 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.513373 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.513597 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.513748 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.514005 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.514137 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.514207 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.514283 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.514307 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:04:14.514413 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.511327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.0.0.1:34510)
0620 08:04:14.511414 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.511428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.511485 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.511660 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.511661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.511662 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:14.511730 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.511732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.512378 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.512380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.513260 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.513264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.513742 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.513743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.513821 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.513844 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.514215 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.514225 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.514271 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.514325 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:04:14.516577 1902 raft_consensus.cc:1261] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Refusing update from remote peer 4085f29540ea4be98ff5aa929a768d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:14.516728 1625 raft_consensus.cc:1261] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Refusing update from remote peer 4085f29540ea4be98ff5aa929a768d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:14.517081 1996 consensus_queue.cc:1035] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:14.517238 1981 consensus_queue.cc:1035] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:14.518536 1982 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: TxnStatusTablet state changed. Reason: New leader 4085f29540ea4be98ff5aa929a768d4b. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.518626 1982 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:14.518646 1978 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: TxnStatusTablet state changed. Reason: New leader 4085f29540ea4be98ff5aa929a768d4b. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.518738 1978 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:14.519629 1981 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.519723 1981 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:14.519793 2001 mvcc.cc:205] Tried to move back new op lower bound from 7298837112896782336 to 7298837112831393792. Current Snapshot: MvccSnapshot[applied={T|T < 7298837112896782336}]
I20260620 08:04:14.519783 1982 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.519861 1982 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4a4a3ea65888419281984db1f94bf67c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:14.520171 2003 mvcc.cc:205] Tried to move back new op lower bound from 7298837112896782336 to 7298837112831393792. Current Snapshot: MvccSnapshot[applied={T|T < 7298837112896782336}]
I20260620 08:04:14.520968 1996 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.521050 1996 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 4085f29540ea4be98ff5aa929a768d4b: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:14.521401 1978 tablet_replica.cc:439] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } } }
I20260620 08:04:14.521485 1978 tablet_replica.cc:442] T 34c0d62c79304c5e84440eafb530f142 P 8011097ac3714be19bc4ed5784274169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:14.521727 2002 mvcc.cc:205] Tried to move back new op lower bound from 7298837112896782336 to 7298837112831393792. Current Snapshot: MvccSnapshot[applied={T|T < 7298837112896782336}]
I20260620 08:04:14.525605 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.522683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.31.115.193:33133)
0620 08:04:14.522735 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.522737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.522796 (+ 59us) server_negotiation.cc:407] Connection header received
0620 08:04:14.522883 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.522883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.522899 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.522956 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:14.523090 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.523573 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.524706 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.524932 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.524996 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.525248 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.525349 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.525411 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.525492 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.525511 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:04:14.531569 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.528455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:49549 (local address 127.31.115.194:45385)
0620 08:04:14.528506 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.528509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.528598 (+ 89us) server_negotiation.cc:407] Connection header received
0620 08:04:14.528632 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.528633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.528645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.528713 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:14.528809 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.529558 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.530599 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.530929 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.530992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.531254 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.531294 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.531352 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.531450 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.531479 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:04:14.531587 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.528376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.31.115.193:49549)
0620 08:04:14.528520 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.528530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.528591 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.528716 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.528717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.528718 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:14.528769 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.528771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.529801 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.529802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.530467 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.530470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.531030 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.531032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.531109 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.531135 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.531381 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.531384 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.531438 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.531496 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260620 08:04:14.534766 1902 raft_consensus.cc:1261] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Refusing update from remote peer 4a4a3ea65888419281984db1f94bf67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:14.534876 1765 raft_consensus.cc:1261] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Refusing update from remote peer 4a4a3ea65888419281984db1f94bf67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:14.535034 1982 consensus_queue.cc:1035] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:14.535208 1984 consensus_queue.cc:1035] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:14.545437 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.542105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33133 (local address 127.31.115.194:47745)
0620 08:04:14.542361 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.542374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.542438 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.542551 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.542552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.542553 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:14.542608 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.542610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.543519 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.543520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.544456 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.544459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.544956 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.544957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.545021 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.545037 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.545270 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.545273 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.545319 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.545360 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":193,"threads_started":1}
I20260620 08:04:14.545434 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.542199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:47745 (local address 127.31.115.193:33133)
0620 08:04:14.542240 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.542243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.542427 (+ 184us) server_negotiation.cc:407] Connection header received
0620 08:04:14.542486 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.542487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.542499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.542552 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:14.542650 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.543413 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.544530 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.544843 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.544916 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.545132 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.545175 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.545229 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.545333 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.545355 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:04:14.555099 1468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:14.555229 1468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:14.556099 1468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Sending DeleteTablet for 3 replicas of tablet 98b1ef9b4a7f4115b005786511a1a523
I20260620 08:04:14.556437 1605 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b1ef9b4a7f4115b005786511a1a523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:14 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:33116
I20260620 08:04:14.556619 1745 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b1ef9b4a7f4115b005786511a1a523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:14 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:34494
I20260620 08:04:14.556783 1885 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b1ef9b4a7f4115b005786511a1a523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:14 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59574
I20260620 08:04:14.557018 2020 tablet_replica.cc:330] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: stopping tablet replica
I20260620 08:04:14.557132 2020 raft_consensus.cc:2229] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:14.557243 2020 raft_consensus.cc:2258] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:14.557390 1468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:14.557557 1468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:14.557753 2020 ts_tablet_manager.cc:1905] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:14.558117 2021 tablet_replica.cc:330] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: stopping tablet replica
I20260620 08:04:14.558290 2021 raft_consensus.cc:2229] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:14.558396 2021 raft_consensus.cc:2258] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:14.558611 2022 tablet_replica.cc:330] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: stopping tablet replica
I20260620 08:04:14.558775 2022 raft_consensus.cc:2229] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:14.558884 2022 raft_consensus.cc:2258] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:14.558917 2021 ts_tablet_manager.cc:1905] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:14.559075 1605 tablet_service.cc:1467] Processing CreateTablet for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa (DEFAULT_TABLE table=test-table [id=e9606c0ad4c9434bb3dade89713d1101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:14.559216 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.559496 2022 ts_tablet_manager.cc:1905] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:14.560581 1885 tablet_service.cc:1467] Processing CreateTablet for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa (DEFAULT_TABLE table=test-table [id=e9606c0ad4c9434bb3dade89713d1101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:14.560693 1977 tablet_bootstrap.cc:492] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Bootstrap starting.
I20260620 08:04:14.561106 1977 tablet_bootstrap.cc:654] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.561579 1745 tablet_service.cc:1467] Processing CreateTablet for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa (DEFAULT_TABLE table=test-table [id=e9606c0ad4c9434bb3dade89713d1101]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:14.561939 1977 tablet_bootstrap.cc:492] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: No bootstrap required, opened a new log
I20260620 08:04:14.561998 1977 ts_tablet_manager.cc:1397] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:14.562158 1977 raft_consensus.cc:348] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.562228 1977 raft_consensus.cc:374] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.562249 1977 raft_consensus.cc:729] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a3ea65888419281984db1f94bf67c, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.562291 1977 consensus_queue.cc:260] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.562374 1977 ts_tablet_manager.cc:1428] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:14.571911 1745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.573372 1976 tablet_bootstrap.cc:492] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Bootstrap starting.
I20260620 08:04:14.573863 1976 tablet_bootstrap.cc:654] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.575026 2021 ts_tablet_manager.cc:1918] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:14.575102 2021 log.cc:1196] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/98b1ef9b4a7f4115b005786511a1a523
I20260620 08:04:14.575373 2021 ts_tablet_manager.cc:1939] T 98b1ef9b4a7f4115b005786511a1a523 P 4085f29540ea4be98ff5aa929a768d4b: Deleting consensus metadata
I20260620 08:04:14.575873 1976 tablet_bootstrap.cc:492] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: No bootstrap required, opened a new log
I20260620 08:04:14.575914 1976 ts_tablet_manager.cc:1397] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 08:04:14.576058 1976 raft_consensus.cc:348] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.576150 1976 raft_consensus.cc:374] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.576172 1976 raft_consensus.cc:729] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.576212 1976 consensus_queue.cc:260] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.576299 1976 ts_tablet_manager.cc:1428] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:14.576785 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:14.578231 1974 tablet_bootstrap.cc:492] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Bootstrap starting.
I20260620 08:04:14.578663 1974 tablet_bootstrap.cc:654] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:14.582075 1974 tablet_bootstrap.cc:492] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: No bootstrap required, opened a new log
I20260620 08:04:14.582151 1974 ts_tablet_manager.cc:1397] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 08:04:14.582306 1974 raft_consensus.cc:348] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.582386 1974 raft_consensus.cc:374] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:14.582407 1974 raft_consensus.cc:729] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8011097ac3714be19bc4ed5784274169, State: Initialized, Role: FOLLOWER
I20260620 08:04:14.582459 1974 consensus_queue.cc:260] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.582548 1974 ts_tablet_manager.cc:1428] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:14.632797 1454 catalog_manager.cc:4841] TS 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385): tablet 98b1ef9b4a7f4115b005786511a1a523 (table test-table [id=086b693e573f4ef9b352a882593767de]) successfully deleted
I20260620 08:04:14.654897 2015 raft_consensus.cc:482] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:14.655032 2015 raft_consensus.cc:504] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.655222 2015 leader_election.cc:288] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:14.655486 1902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5aeb1b02ff4f1aa7d2d9999925caaa" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169" is_pre_election: true
I20260620 08:04:14.655546 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5aeb1b02ff4f1aa7d2d9999925caaa" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c" is_pre_election: true
I20260620 08:04:14.655622 1902 raft_consensus.cc:2454] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 0.
I20260620 08:04:14.655660 1625 raft_consensus.cc:2454] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 0.
I20260620 08:04:14.655845 1696 leader_election.cc:302] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:14.656016 2015 raft_consensus.cc:2781] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:14.656085 2015 raft_consensus.cc:482] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:14.656121 2015 raft_consensus.cc:3037] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.666183 2022 ts_tablet_manager.cc:1918] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:14.666486 2022 log.cc:1196] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/98b1ef9b4a7f4115b005786511a1a523
I20260620 08:04:14.666808 2022 ts_tablet_manager.cc:1939] T 98b1ef9b4a7f4115b005786511a1a523 P 8011097ac3714be19bc4ed5784274169: Deleting consensus metadata
I20260620 08:04:14.667407 1452 catalog_manager.cc:4841] TS 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051): tablet 98b1ef9b4a7f4115b005786511a1a523 (table test-table [id=086b693e573f4ef9b352a882593767de]) successfully deleted
I20260620 08:04:14.667788 2020 ts_tablet_manager.cc:1918] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:14.667953 2020 log.cc:1196] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/98b1ef9b4a7f4115b005786511a1a523
I20260620 08:04:14.668367 2020 ts_tablet_manager.cc:1939] T 98b1ef9b4a7f4115b005786511a1a523 P 4a4a3ea65888419281984db1f94bf67c: Deleting consensus metadata
I20260620 08:04:14.668759 2015 raft_consensus.cc:504] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.668917 2015 leader_election.cc:288] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 election: Requested vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:14.669283 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5aeb1b02ff4f1aa7d2d9999925caaa" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c"
I20260620 08:04:14.669360 1625 raft_consensus.cc:3037] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.670331 1625 raft_consensus.cc:2454] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 1.
I20260620 08:04:14.670568 1696 leader_election.cc:302] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 4a4a3ea65888419281984db1f94bf67c; no voters:
I20260620 08:04:14.670717 2015 raft_consensus.cc:2781] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:14.670760 2015 raft_consensus.cc:686] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 1 LEADER]: Becoming Leader. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Running, Role: LEADER
I20260620 08:04:14.670805 2015 consensus_queue.cc:237] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:14.671761 1902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff5aeb1b02ff4f1aa7d2d9999925caaa" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169"
I20260620 08:04:14.671851 1902 raft_consensus.cc:3037] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:14.672580 1468 catalog_manager.cc:5495] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194). New cstate: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:14.684757 1452 catalog_manager.cc:4841] TS 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133): tablet 98b1ef9b4a7f4115b005786511a1a523 (table test-table [id=086b693e573f4ef9b352a882593767de]) successfully deleted
I20260620 08:04:14.688010 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.685516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.31.115.194:45385)
0620 08:04:14.685572 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.685574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.685581 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:14.685621 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.685622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.685633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.685693 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:14.685805 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.686428 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.687240 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.687445 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.687567 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.687719 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.687757 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.687842 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.687897 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.687925 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:04:14.692991 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.690654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:41515 (local address 127.31.115.194:45385)
0620 08:04:14.690692 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:14.690694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:14.690702 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:14.690758 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:14.690759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:14.690770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:14.690819 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:14.690915 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.691368 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.692256 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.692430 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.692485 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.692711 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.692738 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.692798 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:14.692902 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 08:04:14.692921 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:04:14.692994 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:14.690581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.31.115.194:41515)
0620 08:04:14.690652 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:14.690663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:14.690715 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:14.690838 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:14.690839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:14.690840 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:14.690876 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.690877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.691468 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.691469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.692164 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:14.692166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:14.692533 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:14.692535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:14.692585 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:14.692593 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:14.692833 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:14.692835 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:14.692888 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:04:14.692933 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 08:04:14.695534 1625 raft_consensus.cc:1261] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Refusing update from remote peer 4085f29540ea4be98ff5aa929a768d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:14.695761 2015 consensus_queue.cc:1035] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:04:14.700547 2001 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 08:04:14.763304 1902 raft_consensus.cc:2454] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 1.
I20260620 08:04:14.763623 1903 raft_consensus.cc:1261] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Refusing update from remote peer 4085f29540ea4be98ff5aa929a768d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:14.764269 2015 consensus_queue.cc:1035] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:16.221513 1467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:16.221740 1467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:16.223048 1467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Sending DeleteTablet for 3 replicas of tablet ff5aeb1b02ff4f1aa7d2d9999925caaa
I20260620 08:04:16.223451 1605 tablet_service.cc:1514] Processing DeleteTablet for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:33116
I20260620 08:04:16.223481 1885 tablet_service.cc:1514] Processing DeleteTablet for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59574
I20260620 08:04:16.223745 2053 tablet_replica.cc:330] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: stopping tablet replica
I20260620 08:04:16.223793 1745 tablet_service.cc:1514] Processing DeleteTablet for tablet ff5aeb1b02ff4f1aa7d2d9999925caaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:34494
I20260620 08:04:16.223827 2053 raft_consensus.cc:2229] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:16.223876 2053 raft_consensus.cc:2258] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:16.224190 2053 ts_tablet_manager.cc:1905] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:16.225241 2055 tablet_replica.cc:330] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: stopping tablet replica
I20260620 08:04:16.225241 2054 tablet_replica.cc:330] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: stopping tablet replica
I20260620 08:04:16.225314 2055 raft_consensus.cc:2229] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:16.225314 2054 raft_consensus.cc:2229] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:16.225361 2054 raft_consensus.cc:2258] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:16.225421 2055 raft_consensus.cc:2258] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:16.225688 2054 ts_tablet_manager.cc:1905] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:16.228940 2055 ts_tablet_manager.cc:1905] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 08:04:16 dist-test-slave-c4w8 krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 08:04:16.255031 2053 ts_tablet_manager.cc:1918] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:04:16.255031 2055 ts_tablet_manager.cc:1918] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:04:16.255142 2053 log.cc:1196] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/ff5aeb1b02ff4f1aa7d2d9999925caaa
I20260620 08:04:16.255201 2055 log.cc:1196] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/ff5aeb1b02ff4f1aa7d2d9999925caaa
I20260620 08:04:16.255383 2053 ts_tablet_manager.cc:1939] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4a4a3ea65888419281984db1f94bf67c: Deleting consensus metadata
I20260620 08:04:16.255396 2054 ts_tablet_manager.cc:1918] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:04:16.255533 2054 log.cc:1196] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/ff5aeb1b02ff4f1aa7d2d9999925caaa
I20260620 08:04:16.255542 2055 ts_tablet_manager.cc:1939] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 4085f29540ea4be98ff5aa929a768d4b: Deleting consensus metadata
I20260620 08:04:16.255904 2054 ts_tablet_manager.cc:1939] T ff5aeb1b02ff4f1aa7d2d9999925caaa P 8011097ac3714be19bc4ed5784274169: Deleting consensus metadata
Jun 20 08:04:16 dist-test-slave-c4w8 krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:16.270319 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.256823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.31.115.254:44239)
0620 08:04:16.256988 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.256990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.257004 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:04:16.257110 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.257111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.257125 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.257223 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:16.257934 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.258836 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.259580 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.259926 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.259987 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.266963 (+ 6976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:16.266965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:16.266965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:16.266979 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:16.268904 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:16.269257 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:16.269258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:16.269259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:16.269317 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:16.269489 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:16.269489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:16.269489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:16.269729 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:16.269836 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.270075 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.270173 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260620 08:04:16.270998 1454 catalog_manager.cc:4841] TS 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385): tablet ff5aeb1b02ff4f1aa7d2d9999925caaa (table test-table [id=e9606c0ad4c9434bb3dade89713d1101]) successfully deleted
I20260620 08:04:16.271044 1452 catalog_manager.cc:4841] TS 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051): tablet ff5aeb1b02ff4f1aa7d2d9999925caaa (table test-table [id=e9606c0ad4c9434bb3dade89713d1101]) successfully deleted
I20260620 08:04:16.271394 1452 catalog_manager.cc:4841] TS 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133): tablet ff5aeb1b02ff4f1aa7d2d9999925caaa (table test-table [id=e9606c0ad4c9434bb3dade89713d1101]) successfully deleted
I20260620 08:04:16.272498 1467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:16.272689 1467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:16.274371 1745 tablet_service.cc:1467] Processing CreateTablet for tablet 725b34d76d9f4da1ab8952fe1522714f (DEFAULT_TABLE table=test-table [id=26b10c4a15e24a07bea9ca44097bcec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:16.274369 1605 tablet_service.cc:1467] Processing CreateTablet for tablet 725b34d76d9f4da1ab8952fe1522714f (DEFAULT_TABLE table=test-table [id=26b10c4a15e24a07bea9ca44097bcec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:16.274425 1885 tablet_service.cc:1467] Processing CreateTablet for tablet 725b34d76d9f4da1ab8952fe1522714f (DEFAULT_TABLE table=test-table [id=26b10c4a15e24a07bea9ca44097bcec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:16.274531 1745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b34d76d9f4da1ab8952fe1522714f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:16.274530 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b34d76d9f4da1ab8952fe1522714f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:16.274549 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b34d76d9f4da1ab8952fe1522714f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:16.275647 2064 tablet_bootstrap.cc:492] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Bootstrap starting.
I20260620 08:04:16.275988 2065 tablet_bootstrap.cc:492] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Bootstrap starting.
I20260620 08:04:16.276151 2066 tablet_bootstrap.cc:492] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Bootstrap starting.
I20260620 08:04:16.276170 2064 tablet_bootstrap.cc:654] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:16.276381 2065 tablet_bootstrap.cc:654] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:16.276546 2066 tablet_bootstrap.cc:654] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:16.277014 2064 tablet_bootstrap.cc:492] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: No bootstrap required, opened a new log
I20260620 08:04:16.277072 2064 ts_tablet_manager.cc:1397] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:16.277201 2066 tablet_bootstrap.cc:492] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: No bootstrap required, opened a new log
I20260620 08:04:16.277209 2064 raft_consensus.cc:348] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.277266 2066 ts_tablet_manager.cc:1397] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:16.277292 2064 raft_consensus.cc:374] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:16.277323 2064 raft_consensus.cc:729] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a3ea65888419281984db1f94bf67c, State: Initialized, Role: FOLLOWER
I20260620 08:04:16.277375 2064 consensus_queue.cc:260] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.277392 2066 raft_consensus.cc:348] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.277470 2066 raft_consensus.cc:374] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:16.277498 2066 raft_consensus.cc:729] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Initialized, Role: FOLLOWER
I20260620 08:04:16.277549 2064 ts_tablet_manager.cc:1428] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:16.277567 2066 consensus_queue.cc:260] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.277724 2066 ts_tablet_manager.cc:1428] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:16.278056 2065 tablet_bootstrap.cc:492] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: No bootstrap required, opened a new log
I20260620 08:04:16.278115 2065 ts_tablet_manager.cc:1397] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:16.278266 2065 raft_consensus.cc:348] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.278350 2065 raft_consensus.cc:374] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:16.278383 2065 raft_consensus.cc:729] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8011097ac3714be19bc4ed5784274169, State: Initialized, Role: FOLLOWER
I20260620 08:04:16.278438 2065 consensus_queue.cc:260] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.278628 2065 ts_tablet_manager.cc:1428] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:16.359109 2048 raft_consensus.cc:482] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:16.359221 2048 raft_consensus.cc:504] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.359380 2048 leader_election.cc:288] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:16.359725 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b34d76d9f4da1ab8952fe1522714f" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c" is_pre_election: true
I20260620 08:04:16.359768 1903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b34d76d9f4da1ab8952fe1522714f" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169" is_pre_election: true
I20260620 08:04:16.359848 1625 raft_consensus.cc:2454] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 0.
I20260620 08:04:16.359850 1903 raft_consensus.cc:2454] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 0.
I20260620 08:04:16.360060 1696 leader_election.cc:302] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:16.360208 2048 raft_consensus.cc:2781] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:16.360265 2048 raft_consensus.cc:482] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:16.360281 2048 raft_consensus.cc:3037] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:16.456027 2048 raft_consensus.cc:504] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.456281 2048 leader_election.cc:288] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 election: Requested vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051)
I20260620 08:04:16.456578 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b34d76d9f4da1ab8952fe1522714f" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c"
I20260620 08:04:16.456655 1903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b34d76d9f4da1ab8952fe1522714f" candidate_uuid: "4085f29540ea4be98ff5aa929a768d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8011097ac3714be19bc4ed5784274169"
I20260620 08:04:16.456729 1625 raft_consensus.cc:3037] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:16.456782 1903 raft_consensus.cc:3037] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:16.457676 1625 raft_consensus.cc:2454] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 1.
I20260620 08:04:16.457864 1696 leader_election.cc:302] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 4a4a3ea65888419281984db1f94bf67c; no voters:
I20260620 08:04:16.457950 1903 raft_consensus.cc:2454] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4085f29540ea4be98ff5aa929a768d4b in term 1.
I20260620 08:04:16.457993 2048 raft_consensus.cc:2781] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:16.458041 2048 raft_consensus.cc:686] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 1 LEADER]: Becoming Leader. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Running, Role: LEADER
I20260620 08:04:16.458099 2048 consensus_queue.cc:237] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.458765 1468 catalog_manager.cc:5495] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194). New cstate: current_term: 1 leader_uuid: "4085f29540ea4be98ff5aa929a768d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:16.509418 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.506872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.31.115.194:45385)
0620 08:04:16.507028 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.507030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.507036 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:04:16.507057 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.507058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.507077 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.507152 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:16.507310 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.507853 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.508611 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.508902 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.508975 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.509149 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:16.509184 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:16.509250 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.509304 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.509325 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260620 08:04:16.511579 1625 raft_consensus.cc:1261] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Refusing update from remote peer 4085f29540ea4be98ff5aa929a768d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:16.511571 1903 raft_consensus.cc:1261] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Refusing update from remote peer 4085f29540ea4be98ff5aa929a768d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:16.511822 2048 consensus_queue.cc:1035] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:16.511931 2040 consensus_queue.cc:1035] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:16.523612 1468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:16.523736 1468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:16.524391 1468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Sending DeleteTablet for 3 replicas of tablet 725b34d76d9f4da1ab8952fe1522714f
I20260620 08:04:16.524626 1745 tablet_service.cc:1514] Processing DeleteTablet for tablet 725b34d76d9f4da1ab8952fe1522714f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:34494
I20260620 08:04:16.524712 1605 tablet_service.cc:1514] Processing DeleteTablet for tablet 725b34d76d9f4da1ab8952fe1522714f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:33116
I20260620 08:04:16.524781 2055 tablet_replica.cc:330] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: stopping tablet replica
I20260620 08:04:16.524828 2053 tablet_replica.cc:330] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: stopping tablet replica
I20260620 08:04:16.524857 2055 raft_consensus.cc:2229] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:16.524883 2053 raft_consensus.cc:2229] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:16.524914 2055 raft_consensus.cc:2258] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:16.524923 2053 raft_consensus.cc:2258] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:16.525038 1885 tablet_service.cc:1514] Processing DeleteTablet for tablet 725b34d76d9f4da1ab8952fe1522714f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59574
I20260620 08:04:16.525139 2054 tablet_replica.cc:330] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: stopping tablet replica
I20260620 08:04:16.525096 1468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:04:16.525189 2054 raft_consensus.cc:2229] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:16.525224 2054 raft_consensus.cc:2258] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 08:04:16.525240 1468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:16.525357 2053 ts_tablet_manager.cc:1905] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:16.525509 2055 ts_tablet_manager.cc:1905] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:16.526391 1605 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4c8deb8f874def8a5215b70e3f5fde (DEFAULT_TABLE table=test-table [id=b84a50e9e68c45bb9b45b5fb268d04b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:16.526445 1745 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4c8deb8f874def8a5215b70e3f5fde (DEFAULT_TABLE table=test-table [id=b84a50e9e68c45bb9b45b5fb268d04b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:16.526594 1885 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4c8deb8f874def8a5215b70e3f5fde (DEFAULT_TABLE table=test-table [id=b84a50e9e68c45bb9b45b5fb268d04b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:16.526652 2054 ts_tablet_manager.cc:1905] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:16.526743 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4c8deb8f874def8a5215b70e3f5fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:16.527683 2065 tablet_bootstrap.cc:492] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Bootstrap starting.
I20260620 08:04:16.528053 2065 tablet_bootstrap.cc:654] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:16.528580 2065 tablet_bootstrap.cc:492] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: No bootstrap required, opened a new log
I20260620 08:04:16.528625 2065 ts_tablet_manager.cc:1397] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:16.528749 2065 raft_consensus.cc:348] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.528798 2065 raft_consensus.cc:374] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:16.528846 2065 raft_consensus.cc:729] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8011097ac3714be19bc4ed5784274169, State: Initialized, Role: FOLLOWER
I20260620 08:04:16.528894 2065 consensus_queue.cc:260] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.529008 2065 ts_tablet_manager.cc:1428] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:16.554968 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4c8deb8f874def8a5215b70e3f5fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:16.555794 2054 ts_tablet_manager.cc:1918] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:16.555900 2054 log.cc:1196] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/725b34d76d9f4da1ab8952fe1522714f
I20260620 08:04:16.556136 2054 ts_tablet_manager.cc:1939] T 725b34d76d9f4da1ab8952fe1522714f P 8011097ac3714be19bc4ed5784274169: Deleting consensus metadata
I20260620 08:04:16.556423 1745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4c8deb8f874def8a5215b70e3f5fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:16.556663 2064 tablet_bootstrap.cc:492] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c: Bootstrap starting.
I20260620 08:04:16.557008 2064 tablet_bootstrap.cc:654] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:16.557324 2053 ts_tablet_manager.cc:1918] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:16.557579 2053 log.cc:1196] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/725b34d76d9f4da1ab8952fe1522714f
I20260620 08:04:16.557730 2064 tablet_bootstrap.cc:492] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c: No bootstrap required, opened a new log
I20260620 08:04:16.557785 2064 ts_tablet_manager.cc:1397] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:16.557916 2064 raft_consensus.cc:348] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.558008 2064 raft_consensus.cc:374] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:16.558032 2064 raft_consensus.cc:729] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4a3ea65888419281984db1f94bf67c, State: Initialized, Role: FOLLOWER
I20260620 08:04:16.558045 2066 tablet_bootstrap.cc:492] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Bootstrap starting.
I20260620 08:04:16.558079 2064 consensus_queue.cc:260] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.558183 2064 ts_tablet_manager.cc:1428] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:16.558368 2066 tablet_bootstrap.cc:654] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:16.558575 2053 ts_tablet_manager.cc:1939] T 725b34d76d9f4da1ab8952fe1522714f P 4a4a3ea65888419281984db1f94bf67c: Deleting consensus metadata
I20260620 08:04:16.558995 2066 tablet_bootstrap.cc:492] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: No bootstrap required, opened a new log
I20260620 08:04:16.559049 2066 ts_tablet_manager.cc:1397] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:16.559175 2066 raft_consensus.cc:348] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.559271 2066 raft_consensus.cc:374] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:16.559283 2055 ts_tablet_manager.cc:1918] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:16.559295 2066 raft_consensus.cc:729] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4085f29540ea4be98ff5aa929a768d4b, State: Initialized, Role: FOLLOWER
I20260620 08:04:16.559342 2055 log.cc:1196] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/725b34d76d9f4da1ab8952fe1522714f
I20260620 08:04:16.559367 2066 consensus_queue.cc:260] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.559491 2066 ts_tablet_manager.cc:1428] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:16.559603 2055 ts_tablet_manager.cc:1939] T 725b34d76d9f4da1ab8952fe1522714f P 4085f29540ea4be98ff5aa929a768d4b: Deleting consensus metadata
I20260620 08:04:16.576321 1452 catalog_manager.cc:4841] TS 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051): tablet 725b34d76d9f4da1ab8952fe1522714f (table test-table [id=26b10c4a15e24a07bea9ca44097bcec7]) successfully deleted
I20260620 08:04:16.610373 1452 catalog_manager.cc:4841] TS 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133): tablet 725b34d76d9f4da1ab8952fe1522714f (table test-table [id=26b10c4a15e24a07bea9ca44097bcec7]) successfully deleted
I20260620 08:04:16.643827 1454 catalog_manager.cc:4841] TS 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385): tablet 725b34d76d9f4da1ab8952fe1522714f (table test-table [id=26b10c4a15e24a07bea9ca44097bcec7]) successfully deleted
I20260620 08:04:16.703801 2071 raft_consensus.cc:482] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:16.703907 2071 raft_consensus.cc:504] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.704221 2071 leader_election.cc:288] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385)
I20260620 08:04:16.707628 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.704836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:43623 (local address 127.31.115.194:45385)
0620 08:04:16.704882 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.704884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.704890 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:04:16.704915 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.704916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.704972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.705016 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:16.705232 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.705784 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.706886 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.707262 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.707320 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.707328 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.707509 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.707557 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:04:16.707643 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.704416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.31.115.195:43623)
0620 08:04:16.704572 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:16.704585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:16.704660 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:16.705041 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:16.705042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:16.705046 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:16.705131 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.705133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.705909 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.705910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.706776 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.706778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.707347 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.707348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.707424 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.707428 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:16.707482 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:04:16.707514 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":34,"thread_start_us":92,"threads_started":1}
I20260620 08:04:16.707715 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.704507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:36983 (local address 127.31.115.193:33133)
0620 08:04:16.704672 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.704675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.704807 (+ 132us) server_negotiation.cc:407] Connection header received
0620 08:04:16.704851 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.704852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.704865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.704939 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:16.705154 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.705772 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.706891 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.707323 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.707397 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.707406 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.707577 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.707627 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260620 08:04:16.707862 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4c8deb8f874def8a5215b70e3f5fde" candidate_uuid: "8011097ac3714be19bc4ed5784274169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4085f29540ea4be98ff5aa929a768d4b" is_pre_election: true
I20260620 08:04:16.707921 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.704416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33133 (local address 127.31.115.195:36983)
0620 08:04:16.704716 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:16.704728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:16.704796 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:16.704985 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:16.704986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:16.704989 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:16.705070 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.705072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.705856 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.705857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.706782 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.706784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.707416 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.707417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.707495 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.707499 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:16.707557 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:04:16.707603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":43,"thread_start_us":233,"threads_started":1}
I20260620 08:04:16.707965 1765 raft_consensus.cc:2454] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8011097ac3714be19bc4ed5784274169 in term 0.
I20260620 08:04:16.708151 1838 leader_election.cc:302] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4085f29540ea4be98ff5aa929a768d4b, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:16.708230 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4c8deb8f874def8a5215b70e3f5fde" candidate_uuid: "8011097ac3714be19bc4ed5784274169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c" is_pre_election: true
I20260620 08:04:16.708300 2071 raft_consensus.cc:2781] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:16.708329 1625 raft_consensus.cc:2454] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8011097ac3714be19bc4ed5784274169 in term 0.
I20260620 08:04:16.708343 2071 raft_consensus.cc:482] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:16.708369 2071 raft_consensus.cc:3037] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:16.753773 2071 raft_consensus.cc:504] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.754350 2071 leader_election.cc:288] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [CANDIDATE]: Term 1 election: Requested vote from peers 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133), 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385)
I20260620 08:04:16.754662 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4c8deb8f874def8a5215b70e3f5fde" candidate_uuid: "8011097ac3714be19bc4ed5784274169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4a3ea65888419281984db1f94bf67c"
I20260620 08:04:16.754735 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4c8deb8f874def8a5215b70e3f5fde" candidate_uuid: "8011097ac3714be19bc4ed5784274169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4085f29540ea4be98ff5aa929a768d4b"
I20260620 08:04:16.754848 1765 raft_consensus.cc:3037] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:16.754794 1625 raft_consensus.cc:3037] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:16.755623 1625 raft_consensus.cc:2454] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8011097ac3714be19bc4ed5784274169 in term 1.
I20260620 08:04:16.755872 1765 raft_consensus.cc:2454] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8011097ac3714be19bc4ed5784274169 in term 1.
I20260620 08:04:16.755893 1836 leader_election.cc:302] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4a3ea65888419281984db1f94bf67c, 8011097ac3714be19bc4ed5784274169; no voters:
I20260620 08:04:16.756008 2071 raft_consensus.cc:2781] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:16.756218 2071 raft_consensus.cc:686] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 1 LEADER]: Becoming Leader. State: Replica: 8011097ac3714be19bc4ed5784274169, State: Running, Role: LEADER
I20260620 08:04:16.756321 2071 consensus_queue.cc:237] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } }
I20260620 08:04:16.757247 1468 catalog_manager.cc:5495] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8011097ac3714be19bc4ed5784274169 (127.31.115.195). New cstate: current_term: 1 leader_uuid: "8011097ac3714be19bc4ed5784274169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8011097ac3714be19bc4ed5784274169" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40051 } health_report { overall_health: HEALTHY } } }
I20260620 08:04:16.819480 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.816684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.31.115.195:40051)
0620 08:04:16.816914 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.816917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.816924 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:16.816968 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.816969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.817027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.817099 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:16.817271 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.817902 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.818664 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.818884 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.818935 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.819065 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:16.819173 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:16.819256 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.819292 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.819310 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":125,"threads_started":1}
I20260620 08:04:16.825098 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.822162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:47129 (local address 127.31.115.194:45385)
0620 08:04:16.822210 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.822212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.822333 (+ 121us) server_negotiation.cc:407] Connection header received
0620 08:04:16.822365 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.822366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.822377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.822424 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:16.822724 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.823329 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.824294 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.824500 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.824565 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.824795 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:16.824822 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:16.824866 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.825001 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.825029 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:04:16.825122 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.822039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45385 (local address 127.31.115.195:47129)
0620 08:04:16.822254 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:16.822264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:16.822322 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:16.822541 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:16.822543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:16.822545 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:16.822624 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.822625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.823496 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.823498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.824197 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.824199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.824644 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.824645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.824699 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.824716 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:16.824903 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:16.824930 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 08:04:16.824991 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:04:16.825037 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260620 08:04:16.827759 1765 raft_consensus.cc:1261] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Refusing update from remote peer 8011097ac3714be19bc4ed5784274169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:16.827803 1625 raft_consensus.cc:1261] T 6d4c8deb8f874def8a5215b70e3f5fde P 4a4a3ea65888419281984db1f94bf67c [term 1 FOLLOWER]: Refusing update from remote peer 8011097ac3714be19bc4ed5784274169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:16.828073 2084 consensus_queue.cc:1035] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4a3ea65888419281984db1f94bf67c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:16.828536 2071 consensus_queue.cc:1035] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4085f29540ea4be98ff5aa929a768d4b" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:16.836977 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.834404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:46445 (local address 127.31.115.195:40051)
0620 08:04:16.834451 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:16.834453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:16.834506 (+ 53us) server_negotiation.cc:407] Connection header received
0620 08:04:16.834575 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:16.834577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:16.834589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:16.834659 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:16.834776 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.835366 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.836200 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.836501 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.836567 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.836713 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:16.836748 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:16.836809 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:16.836902 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 08:04:16.836922 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:04:16.837024 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:16.834232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:40051 (local address 127.31.115.194:46445)
0620 08:04:16.834446 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:16.834457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:16.834518 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:16.834664 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:16.834665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:16.834667 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:16.834723 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.834725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.835461 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.835461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.836104 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:16.836107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:16.836570 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:16.836571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:16.836632 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:16.836638 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:16.836847 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:16.836850 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:16.836902 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:04:16.836962 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
W20260620 08:04:16.837450 2036 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 08:04:18.364748 1467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:18.364930 1467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:18.366207 1467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 571b136b05284297b5585f6dc90978fd: Sending DeleteTablet for 3 replicas of tablet 6d4c8deb8f874def8a5215b70e3f5fde
I20260620 08:04:18.366622 1605 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4c8deb8f874def8a5215b70e3f5fde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:18 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:33116
I20260620 08:04:18.366703 1744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4c8deb8f874def8a5215b70e3f5fde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:18 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:34494
I20260620 08:04:18.366677 1885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4c8deb8f874def8a5215b70e3f5fde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:18 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59574
I20260620 08:04:18.366847 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 1540
I20260620 08:04:18.367439 2118 tablet_replica.cc:330] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: stopping tablet replica
I20260620 08:04:18.367511 2118 raft_consensus.cc:2229] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:18.367558 2118 raft_consensus.cc:2258] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:18.367836 2118 ts_tablet_manager.cc:1905] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:18.369509 2118 ts_tablet_manager.cc:1918] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:04:18.369555 2118 log.cc:1196] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/6d4c8deb8f874def8a5215b70e3f5fde
I20260620 08:04:18.369835 2118 ts_tablet_manager.cc:1939] T 6d4c8deb8f874def8a5215b70e3f5fde P 4085f29540ea4be98ff5aa929a768d4b: Deleting consensus metadata
I20260620 08:04:18.370365 1454 catalog_manager.cc:4841] TS 4085f29540ea4be98ff5aa929a768d4b (127.31.115.194:45385): tablet 6d4c8deb8f874def8a5215b70e3f5fde (table test-table [id=b84a50e9e68c45bb9b45b5fb268d04b8]) successfully deleted
I20260620 08:04:18.373884 2119 tablet_replica.cc:330] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: stopping tablet replica
I20260620 08:04:18.373992 2119 raft_consensus.cc:2229] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:18.374131 2119 raft_consensus.cc:2258] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:18.374452 2119 ts_tablet_manager.cc:1905] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:04:18.375530 1452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:33133 (error 108)
I20260620 08:04:18.375849 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 1680
I20260620 08:04:18.375996 2119 ts_tablet_manager.cc:1918] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:04:18.376053 2119 log.cc:1196] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/6d4c8deb8f874def8a5215b70e3f5fde
I20260620 08:04:18.376423 2119 ts_tablet_manager.cc:1939] T 6d4c8deb8f874def8a5215b70e3f5fde P 8011097ac3714be19bc4ed5784274169: Deleting consensus metadata
I20260620 08:04:18.376828 1452 catalog_manager.cc:4841] TS 8011097ac3714be19bc4ed5784274169 (127.31.115.195:40051): tablet 6d4c8deb8f874def8a5215b70e3f5fde (table test-table [id=b84a50e9e68c45bb9b45b5fb268d04b8]) successfully deleted
I20260620 08:04:18.381932 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:18.375969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33133 (local address 127.0.0.1:57972)
0620 08:04:18.381803 (+ 5834us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:18.381887 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":61,"threads_started":1}
W20260620 08:04:18.382093 1452 catalog_manager.cc:4584] TS 4a4a3ea65888419281984db1f94bf67c (127.31.115.193:33133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d4c8deb8f874def8a5215b70e3f5fde: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33133: connect: Connection refused (error 111)
I20260620 08:04:18.383435 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 1820
I20260620 08:04:18.389262 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 1436
2026-06-20T08:04:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6054 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:18 dist-test-slave-c4w8 krb5kdc[2124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:18 dist-test-slave-c4w8 krb5kdc[2124](info): set up 2 sockets
Jun 20 08:04:18 dist-test-slave-c4w8 krb5kdc[2124](info): commencing operation
krb5kdc: starting...
W20260620 08:04:19.442245 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:19 dist-test-slave-c4w8 krb5kdc[2124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:19.570147 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:43059
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:43059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:19.653726 2137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:19.653947 2137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:19.653971 2137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:19.655298 2137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:19.655359 2137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:19.655375 2137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:19.655387 2137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:19.655400 2137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:19.657073 2137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:43059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:19.657289 2137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:19.657472 2137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:19.662566 2143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:19.662690 2142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:19.662750 2137 server_base.cc:1034] running on GCE node
W20260620 08:04:19.662567 2145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:19.663111 2137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:19.663467 2137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:19.664665 2137 hybrid_clock.cc:648] HybridClock initialized: now 1781942659664629 us; error 48 us; skew 500 ppm
Jun 20 08:04:19 dist-test-slave-c4w8 krb5kdc[2124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942659, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:19.667711 2137 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:19.668228 2137 webserver.cc:460] Webserver started at http://127.31.115.254:35977/ using document root <none> and password file <none>
I20260620 08:04:19.668468 2137 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:19.668538 2137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:19.668695 2137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:19.669874 2137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "5c04ec1fab77420c9f1241da303563fd"
format_stamp: "Formatted at 2026-06-20 08:04:19 on dist-test-slave-c4w8"
server_key: "b75ec82c33d94b0a388c2b9c7c60710e"
server_key_iv: "96449827369d6cfb3b8198bd560937f6"
server_key_version: "encryptionkey@0"
I20260620 08:04:19.670250 2137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "5c04ec1fab77420c9f1241da303563fd"
format_stamp: "Formatted at 2026-06-20 08:04:19 on dist-test-slave-c4w8"
server_key: "b75ec82c33d94b0a388c2b9c7c60710e"
server_key_iv: "96449827369d6cfb3b8198bd560937f6"
server_key_version: "encryptionkey@0"
I20260620 08:04:19.671664 2137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:19.672695 2152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:19.672948 2137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:19.673089 2137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "5c04ec1fab77420c9f1241da303563fd"
format_stamp: "Formatted at 2026-06-20 08:04:19 on dist-test-slave-c4w8"
server_key: "b75ec82c33d94b0a388c2b9c7c60710e"
server_key_iv: "96449827369d6cfb3b8198bd560937f6"
server_key_version: "encryptionkey@0"
I20260620 08:04:19.673178 2137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:19.687175 2137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:19.689872 2137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:19.690055 2137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:19.695276 2137 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:43059
I20260620 08:04:19.695312 2204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:43059 every 8 connection(s)
I20260620 08:04:19.695716 2137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:19.697019 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:19.699564 2205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd: Bootstrap starting.
I20260620 08:04:19.700227 2205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:19.700565 2205 log.cc:824] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:19.701383 2205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd: No bootstrap required, opened a new log
I20260620 08:04:19.703136 2205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04ec1fab77420c9f1241da303563fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43059 } }
I20260620 08:04:19.703311 2205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:19.703337 2205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c04ec1fab77420c9f1241da303563fd, State: Initialized, Role: FOLLOWER
I20260620 08:04:19.703509 2205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04ec1fab77420c9f1241da303563fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43059 } }
I20260620 08:04:19.703590 2205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:19.703634 2205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:19.703687 2205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:19.704280 2205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04ec1fab77420c9f1241da303563fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43059 } }
I20260620 08:04:19.704383 2205 leader_election.cc:302] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c04ec1fab77420c9f1241da303563fd; no voters:
I20260620 08:04:19.704624 2205 leader_election.cc:288] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:19.704774 2208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:19.704937 2205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:19.705271 2208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [term 1 LEADER]: Becoming Leader. State: Replica: 5c04ec1fab77420c9f1241da303563fd, State: Running, Role: LEADER
I20260620 08:04:19.705394 2208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04ec1fab77420c9f1241da303563fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43059 } }
I20260620 08:04:19.705456 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2137
I20260620 08:04:19.705549 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:19.705772 32207 external_mini_cluster.cc:1428] Setting key 9d74e20619f3612012a601b6564a5b24
I20260620 08:04:19.706030 2209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c04ec1fab77420c9f1241da303563fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04ec1fab77420c9f1241da303563fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43059 } } }
I20260620 08:04:19.706122 2209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:19.706563 2213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:19.707223 2213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:19.706548 2210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c04ec1fab77420c9f1241da303563fd. Latest consensus state: current_term: 1 leader_uuid: "5c04ec1fab77420c9f1241da303563fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04ec1fab77420c9f1241da303563fd" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43059 } } }
I20260620 08:04:19.707787 2210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:19.708986 2213 catalog_manager.cc:1347] Generated new cluster ID: c985412aa2074416b11a3c79def12596
I20260620 08:04:19.709039 2213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:19 dist-test-slave-c4w8 krb5kdc[2124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:19.718684 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:19.706290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.31.115.254:43059)
0620 08:04:19.707508 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:19.707519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:19.707553 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:04:19.707890 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:19.707901 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:19.707974 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:19.708137 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:19.709003 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:19.710080 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:19.710909 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:19.711346 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:19.711464 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:19.714045 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:19.714057 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:19.714058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:19.714071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:19.716444 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:19.716999 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:19.717010 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:19.717010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:19.717277 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:19.717567 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:19.717568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:19.717569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:19.717984 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:19.718159 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:19.718288 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:04:19.718448 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":65,"threads_started":1}
I20260620 08:04:19.727339 2213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:19.728008 2213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:19.730937 2213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c04ec1fab77420c9f1241da303563fd: Generated new TSK 0
I20260620 08:04:19.731212 2213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:19.765195 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:43059
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:19.868824 2231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:19.869040 2231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:19.869067 2231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:19.870409 2231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:19.870462 2231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:19.870493 2231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:19.872017 2231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:43059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:19.872273 2231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:19.872531 2231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:19.879256 2236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:19.879334 2239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:19.879292 2237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:19.879566 2231 server_base.cc:1034] running on GCE node
I20260620 08:04:19.879840 2231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:19.880108 2231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:19.881340 2231 hybrid_clock.cc:648] HybridClock initialized: now 1781942659881320 us; error 40 us; skew 500 ppm
Jun 20 08:04:19 dist-test-slave-c4w8 krb5kdc[2124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942659, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:19.884182 2231 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:19.884694 2231 webserver.cc:460] Webserver started at http://127.31.115.193:33379/ using document root <none> and password file <none>
I20260620 08:04:19.884963 2231 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:19.885033 2231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:19.885183 2231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:19.886389 2231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "ca7025bc58594d7cb841e0895331db91"
format_stamp: "Formatted at 2026-06-20 08:04:19 on dist-test-slave-c4w8"
server_key: "cde4ed2f88a579ae95b3c0fbe61318ee"
server_key_iv: "434e8a00589f5151c0bf823605702d8d"
server_key_version: "encryptionkey@0"
I20260620 08:04:19.886768 2231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "ca7025bc58594d7cb841e0895331db91"
format_stamp: "Formatted at 2026-06-20 08:04:19 on dist-test-slave-c4w8"
server_key: "cde4ed2f88a579ae95b3c0fbe61318ee"
server_key_iv: "434e8a00589f5151c0bf823605702d8d"
server_key_version: "encryptionkey@0"
I20260620 08:04:19.888226 2231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:04:19.889266 2246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:19.889559 2231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:19.889746 2231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "ca7025bc58594d7cb841e0895331db91"
format_stamp: "Formatted at 2026-06-20 08:04:19 on dist-test-slave-c4w8"
server_key: "cde4ed2f88a579ae95b3c0fbe61318ee"
server_key_iv: "434e8a00589f5151c0bf823605702d8d"
server_key_version: "encryptionkey@0"
I20260620 08:04:19.889839 2231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:19.917997 2231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:19.920646 2231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:19.920866 2231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:19.921159 2231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:19.921509 2231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:19.921566 2231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:19.921602 2231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:19.921643 2231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:19.927899 2231 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:38841
I20260620 08:04:19.928221 2359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:38841 every 8 connection(s)
I20260620 08:04:19.928308 2231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:19.931787 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2231
I20260620 08:04:19.931892 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:19.932070 32207 external_mini_cluster.cc:1428] Setting key e7cec705a28f5384bf99ead1cc3932c4
Jun 20 08:04:19 dist-test-slave-c4w8 krb5kdc[2124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942659, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:19.939636 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:19.929168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:55465 (local address 127.31.115.254:43059)
0620 08:04:19.929257 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:19.929260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:19.929744 (+ 484us) server_negotiation.cc:407] Connection header received
0620 08:04:19.930312 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:19.930314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:19.930329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:19.930389 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:19.931417 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:19.932214 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:19.933499 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:19.933871 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:19.933951 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:19.936455 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:19.936457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:19.936457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:19.936466 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:19.938086 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:19.938545 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:19.938546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:19.938546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:19.938607 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:19.938935 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:19.938936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:19.938937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:19.939063 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:19.939138 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:19.939442 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 08:04:19.939537 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:04:19.939862 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:19.929133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:43059 (local address 127.31.115.193:55465)
0620 08:04:19.929347 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:19.929397 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:19.930174 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:19.930473 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:19.930477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:19.931108 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:19.931309 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:19.931315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:19.932335 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:19.932336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:19.933392 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:19.933395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:19.933963 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:19.933964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:19.934090 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:19.934711 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:19.934723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:19.936354 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:19.938197 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:19.938208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:19.938210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:19.938456 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:19.938723 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:19.938724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:19.938725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:19.938846 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:19.939141 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:19.939150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:19.939305 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 08:04:19.939419 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:04:19.939595 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260620 08:04:19.940418 2360 heartbeater.cc:347] Connected to a master server at 127.31.115.254:43059
I20260620 08:04:19.940518 2360 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:19.940737 2360 heartbeater.cc:511] Master 127.31.115.254:43059 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:04:19.941325 2169 ts_manager.cc:194] Registered new tserver with Master: ca7025bc58594d7cb841e0895331db91 (127.31.115.193:38841)
I20260620 08:04:19.942070 2169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:55465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:19.960398 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:43059
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:20.051826 2365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:20.052065 2365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:20.052091 2365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:20.053244 2365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:20.053284 2365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:20.053311 2365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:20.054821 2365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:43059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:20.055047 2365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:20.055246 2365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:20.061070 2373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:20.061097 2371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:20.061070 2370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:20.061714 2365 server_base.cc:1034] running on GCE node
I20260620 08:04:20.062012 2365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:20.062376 2365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:20.063576 2365 hybrid_clock.cc:648] HybridClock initialized: now 1781942660063574 us; error 47 us; skew 500 ppm
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942660, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:20.066569 2365 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:20.067121 2365 webserver.cc:460] Webserver started at http://127.31.115.194:34467/ using document root <none> and password file <none>
I20260620 08:04:20.067387 2365 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:20.067462 2365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:20.067631 2365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:20.068759 2365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "de1357a6e32b4dd5b11c94992d30d327"
format_stamp: "Formatted at 2026-06-20 08:04:20 on dist-test-slave-c4w8"
server_key: "2c09be968b7699e8e68a058cd89871a5"
server_key_iv: "639c02b44340f29cf235789c2260c152"
server_key_version: "encryptionkey@0"
I20260620 08:04:20.069159 2365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "de1357a6e32b4dd5b11c94992d30d327"
format_stamp: "Formatted at 2026-06-20 08:04:20 on dist-test-slave-c4w8"
server_key: "2c09be968b7699e8e68a058cd89871a5"
server_key_iv: "639c02b44340f29cf235789c2260c152"
server_key_version: "encryptionkey@0"
I20260620 08:04:20.070760 2365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 08:04:20.071740 2380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:20.072021 2365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:20.072182 2365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "de1357a6e32b4dd5b11c94992d30d327"
format_stamp: "Formatted at 2026-06-20 08:04:20 on dist-test-slave-c4w8"
server_key: "2c09be968b7699e8e68a058cd89871a5"
server_key_iv: "639c02b44340f29cf235789c2260c152"
server_key_version: "encryptionkey@0"
I20260620 08:04:20.072290 2365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:20.091521 2365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:20.094005 2365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:20.094178 2365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:20.094444 2365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:20.094825 2365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:20.094887 2365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:20.094935 2365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:20.094959 2365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:20.101231 2365 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:42133
I20260620 08:04:20.101343 2493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:42133 every 8 connection(s)
I20260620 08:04:20.101703 2365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:20.106252 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2365
I20260620 08:04:20.106374 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:20.106596 32207 external_mini_cluster.cc:1428] Setting key 062394bca15cb3c2cca02fa6f2b25b8f
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942660, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:20.113238 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:20.102593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:36609 (local address 127.31.115.254:43059)
0620 08:04:20.102698 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:20.102701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:20.102883 (+ 182us) server_negotiation.cc:407] Connection header received
0620 08:04:20.103367 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:20.103368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:20.103379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:20.103444 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:20.104549 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.105089 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.106416 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.106658 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.106752 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:20.109226 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:20.109228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:20.109228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:20.109237 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:20.111473 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:20.111973 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:20.111975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:20.111975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:20.112048 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:20.112474 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:20.112475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:20.112475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:20.112650 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:20.112748 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:20.113018 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 08:04:20.113125 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 08:04:20.113432 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:20.102556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:43059 (local address 127.31.115.194:36609)
0620 08:04:20.102721 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:20.102765 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:20.103261 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:20.103509 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:20.103514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:20.104141 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:20.104431 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:20.104437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.105204 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.105205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:20.106312 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:20.106316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.106766 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.106768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:20.106926 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:20.107595 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:20.107605 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:20.109119 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:20.111592 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:20.111600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:20.111602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:20.111888 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:20.112280 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:20.112281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:20.112281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:20.112395 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:20.112737 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:20.112743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:20.112877 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 08:04:20.112973 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 08:04:20.113125 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 08:04:20.114163 2494 heartbeater.cc:347] Connected to a master server at 127.31.115.254:43059
I20260620 08:04:20.114285 2494 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:20.114492 2494 heartbeater.cc:511] Master 127.31.115.254:43059 requested a full tablet report, sending...
I20260620 08:04:20.114953 2169 ts_manager.cc:194] Registered new tserver with Master: de1357a6e32b4dd5b11c94992d30d327 (127.31.115.194:42133)
I20260620 08:04:20.115382 2169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:36609
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:20.134778 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:43059
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:20.220873 2499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:20.221098 2499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:20.221124 2499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:20.222443 2499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:20.222496 2499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:20.222527 2499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:20.224097 2499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:43059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:20.224380 2499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:20.224638 2499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:20.230202 2505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:20.230268 2499 server_base.cc:1034] running on GCE node
W20260620 08:04:20.230170 2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:20.230369 2504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:20.230584 2499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:20.230893 2499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:20.232064 2499 hybrid_clock.cc:648] HybridClock initialized: now 1781942660232048 us; error 48 us; skew 500 ppm
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942660, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:20.234897 2499 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:20.235442 2499 webserver.cc:460] Webserver started at http://127.31.115.195:39329/ using document root <none> and password file <none>
I20260620 08:04:20.235695 2499 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:20.235744 2499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:20.235889 2499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:20.237007 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "846ec28a59a34cae8a40dc2780372164"
format_stamp: "Formatted at 2026-06-20 08:04:20 on dist-test-slave-c4w8"
server_key: "ab98800ba7d7275ef679734d5393644f"
server_key_iv: "4a034dd53f6a7b989cdfc463601966d7"
server_key_version: "encryptionkey@0"
I20260620 08:04:20.237375 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "846ec28a59a34cae8a40dc2780372164"
format_stamp: "Formatted at 2026-06-20 08:04:20 on dist-test-slave-c4w8"
server_key: "ab98800ba7d7275ef679734d5393644f"
server_key_iv: "4a034dd53f6a7b989cdfc463601966d7"
server_key_version: "encryptionkey@0"
I20260620 08:04:20.239109 2499 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:04:20.240264 2514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:20.240546 2499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:20.240739 2499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "846ec28a59a34cae8a40dc2780372164"
format_stamp: "Formatted at 2026-06-20 08:04:20 on dist-test-slave-c4w8"
server_key: "ab98800ba7d7275ef679734d5393644f"
server_key_iv: "4a034dd53f6a7b989cdfc463601966d7"
server_key_version: "encryptionkey@0"
I20260620 08:04:20.240831 2499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:20.248209 2499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:20.250805 2499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:20.250982 2499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:20.251241 2499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:20.251577 2499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:20.251638 2499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:20.251680 2499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:20.251724 2499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:20.258014 2499 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:37009
I20260620 08:04:20.258122 2627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:37009 every 8 connection(s)
I20260620 08:04:20.258430 2499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:20.260527 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2499
I20260620 08:04:20.260695 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:20.260888 32207 external_mini_cluster.cc:1428] Setting key 81b2aa218dfd0d74dc53596779b94e65
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942660, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:20.269276 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:20.259343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:48849 (local address 127.31.115.254:43059)
0620 08:04:20.259426 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:20.259429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:20.259663 (+ 234us) server_negotiation.cc:407] Connection header received
0620 08:04:20.260215 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:20.260217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:20.260229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:20.260287 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:20.261411 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.262172 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.263273 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.263495 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.263570 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:20.266025 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:20.266026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:20.266027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:20.266035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:20.267703 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:20.268147 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:20.268149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:20.268149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:20.268210 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:20.268523 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:20.268524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:20.268525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:20.268667 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:20.268737 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:20.269127 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 08:04:20.269209 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:04:20.269438 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:20.259284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:43059 (local address 127.31.115.195:48849)
0620 08:04:20.259497 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:20.259548 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:20.260095 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:20.260346 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:20.260350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:20.261116 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:20.261331 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:20.261338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.262277 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.262278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:20.263130 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:20.263162 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:20.263600 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:20.263601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:20.263728 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:20.264283 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:20.264292 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:20.265922 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:20.267804 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:20.267811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:20.267813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:20.268044 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:20.268305 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:20.268306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:20.268306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:20.268429 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:20.268746 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:20.268751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:20.268888 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 08:04:20.269018 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 08:04:20.269184 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
I20260620 08:04:20.270186 2628 heartbeater.cc:347] Connected to a master server at 127.31.115.254:43059
I20260620 08:04:20.270332 2628 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:20.270542 2628 heartbeater.cc:511] Master 127.31.115.254:43059 requested a full tablet report, sending...
I20260620 08:04:20.271075 2169 ts_manager.cc:194] Registered new tserver with Master: 846ec28a59a34cae8a40dc2780372164 (127.31.115.195:37009)
I20260620 08:04:20.271486 2169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:48849
I20260620 08:04:20.272907 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:04:20.284864 2215 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:20.283823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.31.115.254:43059)
0620 08:04:20.283897 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:20.283899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:20.284016 (+ 117us) server_negotiation.cc:407] Connection header received
0620 08:04:20.284093 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:20.284094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:20.284103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:20.284165 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:20.284755 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57198: BlockingRecv error: recv got EOF from 127.0.0.1:57198 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:04:20.286195 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2231
I20260620 08:04:20.291874 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2365
I20260620 08:04:20.296905 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2499
I20260620 08:04:20.302285 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2137
2026-06-20T08:04:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1912 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2640](info): set up 2 sockets
Jun 20 08:04:20 dist-test-slave-c4w8 krb5kdc[2640](info): commencing operation
krb5kdc: starting...
W20260620 08:04:21.357874 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:21 dist-test-slave-c4w8 krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:21.482216 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38287
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:21.565579 2653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:21.565912 2653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:21.565968 2653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:21.567970 2653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:21.568027 2653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:21.568060 2653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:21.568084 2653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:21.568112 2653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:21.569926 2653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:21.570199 2653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:21.570437 2653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:21.575809 2658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:21.575786 2659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:21.575884 2661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:21.575886 2653 server_base.cc:1034] running on GCE node
I20260620 08:04:21.576328 2653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:21.576635 2653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:21.577788 2653 hybrid_clock.cc:648] HybridClock initialized: now 1781942661577711 us; error 95 us; skew 500 ppm
Jun 20 08:04:21 dist-test-slave-c4w8 krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942661, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:21.585240 2653 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:21.585840 2653 webserver.cc:460] Webserver started at http://127.31.115.254:44183/ using document root <none> and password file <none>
I20260620 08:04:21.586082 2653 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:21.586153 2653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:21.586314 2653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:21.587642 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "ca13ffe771ec445eb811542404b7ffc8"
format_stamp: "Formatted at 2026-06-20 08:04:21 on dist-test-slave-c4w8"
server_key: "4a16b78114651fec4eea8d28ba52e446"
server_key_iv: "5da262afd5618588060ca415df1bd265"
server_key_version: "encryptionkey@0"
I20260620 08:04:21.588018 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "ca13ffe771ec445eb811542404b7ffc8"
format_stamp: "Formatted at 2026-06-20 08:04:21 on dist-test-slave-c4w8"
server_key: "4a16b78114651fec4eea8d28ba52e446"
server_key_iv: "5da262afd5618588060ca415df1bd265"
server_key_version: "encryptionkey@0"
I20260620 08:04:21.589452 2653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:21.590400 2668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:21.590631 2653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:21.590742 2653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "ca13ffe771ec445eb811542404b7ffc8"
format_stamp: "Formatted at 2026-06-20 08:04:21 on dist-test-slave-c4w8"
server_key: "4a16b78114651fec4eea8d28ba52e446"
server_key_iv: "5da262afd5618588060ca415df1bd265"
server_key_version: "encryptionkey@0"
I20260620 08:04:21.590831 2653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:21.600111 2653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:21.602326 2653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:21.602502 2653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:21.606739 2653 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:38287
I20260620 08:04:21.606772 2720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:38287 every 8 connection(s)
I20260620 08:04:21.607286 2653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:21.607465 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2653
I20260620 08:04:21.607565 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:21.607776 32207 external_mini_cluster.cc:1428] Setting key 603c9dab3e4f35c664c0a7029078ce6c
I20260620 08:04:21.608199 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:21.611665 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8: Bootstrap starting.
I20260620 08:04:21.612430 2721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:21.612738 2721 log.cc:824] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:21.613698 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8: No bootstrap required, opened a new log
Jun 20 08:04:21 dist-test-slave-c4w8 krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:21.615779 2721 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca13ffe771ec445eb811542404b7ffc8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38287 } }
I20260620 08:04:21.615952 2721 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:21.615998 2721 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca13ffe771ec445eb811542404b7ffc8, State: Initialized, Role: FOLLOWER
I20260620 08:04:21.616137 2721 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca13ffe771ec445eb811542404b7ffc8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38287 } }
I20260620 08:04:21.616241 2721 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:21.616286 2721 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:21.616338 2721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:21.616902 2721 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca13ffe771ec445eb811542404b7ffc8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38287 } }
I20260620 08:04:21.617028 2721 leader_election.cc:302] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca13ffe771ec445eb811542404b7ffc8; no voters:
I20260620 08:04:21.617249 2721 leader_election.cc:288] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:21.617283 2726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:21.617404 2726 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [term 1 LEADER]: Becoming Leader. State: Replica: ca13ffe771ec445eb811542404b7ffc8, State: Running, Role: LEADER
I20260620 08:04:21.617537 2721 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:21.617524 2726 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca13ffe771ec445eb811542404b7ffc8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38287 } }
I20260620 08:04:21.617962 2726 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca13ffe771ec445eb811542404b7ffc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca13ffe771ec445eb811542404b7ffc8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38287 } } }
I20260620 08:04:21.618069 2726 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:21.618309 2726 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca13ffe771ec445eb811542404b7ffc8. Latest consensus state: current_term: 1 leader_uuid: "ca13ffe771ec445eb811542404b7ffc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca13ffe771ec445eb811542404b7ffc8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38287 } } }
I20260620 08:04:21.618417 2726 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:21.618558 2733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:21.619112 2733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:21.620401 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:21.608347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.31.115.254:38287)
0620 08:04:21.608596 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:21.608606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:21.608639 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:04:21.609415 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:21.609425 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:21.609547 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:21.609734 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:21.610352 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:21.611743 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:21.612871 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:21.613549 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:21.613754 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:21.616114 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:21.616119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:21.616120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:21.616133 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:21.618242 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:21.618716 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:21.618719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:21.618719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:21.618837 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:21.619070 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:21.619071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:21.619072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:21.619424 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:21.619587 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:21.620049 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 08:04:21.620198 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260620 08:04:21.620960 2733 catalog_manager.cc:1347] Generated new cluster ID: 7ededcdd7ffc4b889f98acee72753a24
I20260620 08:04:21.621011 2733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:21.621384 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:04:21.636161 2733 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:21.636793 2733 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:21.647950 2733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca13ffe771ec445eb811542404b7ffc8: Generated new TSK 0
I20260620 08:04:21.648216 2733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:21.651065 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:21.646125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.31.115.254:38287)
0620 08:04:21.646183 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:21.646186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:21.646416 (+ 230us) server_negotiation.cc:407] Connection header received
0620 08:04:21.646489 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:21.646490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:21.646503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:21.646570 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:21.647085 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:21.647946 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:21.648871 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:21.649236 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:21.649316 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:21.650141 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:21.650143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:21.650143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:21.650562 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:21.650717 (+ 155us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:21.650734 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:21.650813 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:21.650924 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 08:04:21.650960 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:04:21.654094 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2653
2026-06-20T08:04:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1350 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:21 dist-test-slave-c4w8 krb5kdc[2754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:21 dist-test-slave-c4w8 krb5kdc[2754](info): set up 2 sockets
Jun 20 08:04:21 dist-test-slave-c4w8 krb5kdc[2754](info): commencing operation
krb5kdc: starting...
W20260620 08:04:22.701994 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:22 dist-test-slave-c4w8 krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:22.824999 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38121
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:22.910981 2767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:22.911209 2767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:22.911233 2767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:22.912425 2767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:22.912472 2767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:22.912487 2767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:22.912499 2767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:22.912513 2767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:22.914141 2767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:22.914424 2767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:22.914673 2767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:22.920106 2773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:22.920117 2772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:22.920117 2775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:22.920473 2767 server_base.cc:1034] running on GCE node
I20260620 08:04:22.920795 2767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:22.921221 2767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:22.922443 2767 hybrid_clock.cc:648] HybridClock initialized: now 1781942662922387 us; error 75 us; skew 500 ppm
Jun 20 08:04:22 dist-test-slave-c4w8 krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942662, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:22.925524 2767 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:22.926177 2767 webserver.cc:460] Webserver started at http://127.31.115.254:35029/ using document root <none> and password file <none>
I20260620 08:04:22.926482 2767 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:22.926570 2767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:22.926759 2767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:22.928332 2767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "51b7de0d7df340e5af03af31c510b45a"
format_stamp: "Formatted at 2026-06-20 08:04:22 on dist-test-slave-c4w8"
server_key: "1110203aa5214bd7714aed19c00ef554"
server_key_iv: "c5c2c6a381d6dd37477d644e4b7fc47f"
server_key_version: "encryptionkey@0"
I20260620 08:04:22.928817 2767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "51b7de0d7df340e5af03af31c510b45a"
format_stamp: "Formatted at 2026-06-20 08:04:22 on dist-test-slave-c4w8"
server_key: "1110203aa5214bd7714aed19c00ef554"
server_key_iv: "c5c2c6a381d6dd37477d644e4b7fc47f"
server_key_version: "encryptionkey@0"
I20260620 08:04:22.930787 2767 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260620 08:04:22.931877 2782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:22.932159 2767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:22.932296 2767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "51b7de0d7df340e5af03af31c510b45a"
format_stamp: "Formatted at 2026-06-20 08:04:22 on dist-test-slave-c4w8"
server_key: "1110203aa5214bd7714aed19c00ef554"
server_key_iv: "c5c2c6a381d6dd37477d644e4b7fc47f"
server_key_version: "encryptionkey@0"
I20260620 08:04:22.932395 2767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:22.966360 2767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:22.969095 2767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:22.969282 2767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:22.973896 2767 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:38121
I20260620 08:04:22.973974 2834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:38121 every 8 connection(s)
I20260620 08:04:22.974380 2767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:22.975606 2835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:22.978135 2835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a: Bootstrap starting.
I20260620 08:04:22.978832 2835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:22.979143 2835 log.cc:824] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:22.979933 2835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a: No bootstrap required, opened a new log
I20260620 08:04:22.981127 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2767
I20260620 08:04:22.981283 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:22.981478 32207 external_mini_cluster.cc:1428] Setting key 3b3a0a108f0b61fd5b60c733ea24df7e
I20260620 08:04:22.981971 2835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b7de0d7df340e5af03af31c510b45a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38121 } }
I20260620 08:04:22.982203 2835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:22.982270 2835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b7de0d7df340e5af03af31c510b45a, State: Initialized, Role: FOLLOWER
I20260620 08:04:22.982427 2835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b7de0d7df340e5af03af31c510b45a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38121 } }
I20260620 08:04:22.982520 2835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:22.982550 2835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:22.982599 2835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:04:22 dist-test-slave-c4w8 krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:22.992587 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:22.982168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.31.115.254:38121)
0620 08:04:22.982415 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:22.982425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:22.982461 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:04:22.982599 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:22.982607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:22.982686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:22.982845 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:22.983560 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:22.984551 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:22.985353 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:22.985894 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:22.985989 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:22.988641 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:22.988645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:22.988645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:22.988659 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:22.990638 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:22.991078 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:22.991082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:22.991082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:22.991202 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:22.991475 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:22.991477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:22.991477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:22.991801 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:22.991919 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:22.992086 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 08:04:22.992349 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":131,"threads_started":1}
I20260620 08:04:23.019677 2835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b7de0d7df340e5af03af31c510b45a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38121 } }
I20260620 08:04:23.019908 2835 leader_election.cc:302] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b7de0d7df340e5af03af31c510b45a; no voters:
I20260620 08:04:23.020187 2835 leader_election.cc:288] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:23.020292 2840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:23.020501 2840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [term 1 LEADER]: Becoming Leader. State: Replica: 51b7de0d7df340e5af03af31c510b45a, State: Running, Role: LEADER
I20260620 08:04:23.020596 2835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:23.020665 2840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b7de0d7df340e5af03af31c510b45a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38121 } }
I20260620 08:04:23.021123 2841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b7de0d7df340e5af03af31c510b45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b7de0d7df340e5af03af31c510b45a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38121 } } }
I20260620 08:04:23.021214 2841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:23.021504 2842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b7de0d7df340e5af03af31c510b45a. Latest consensus state: current_term: 1 leader_uuid: "51b7de0d7df340e5af03af31c510b45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b7de0d7df340e5af03af31c510b45a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38121 } } }
I20260620 08:04:23.021865 2842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:23.021932 2845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:23.022600 2845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:23.026057 2845 catalog_manager.cc:1347] Generated new cluster ID: eef491e7ee43466ea8607a0f6f790707
I20260620 08:04:23.026125 2845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:23.039667 2845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:23.040621 2845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:23.052332 2845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b7de0d7df340e5af03af31c510b45a: Generated new TSK 0
I20260620 08:04:23.052608 2845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
W20260620 08:04:23.904688 32207 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.115.193: real 0.846s user 0.003s sys 0.000s
I20260620 08:04:23.905120 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38121
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:23.990017 2861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:23.990223 2861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:23.990247 2861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:23.991381 2861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:23.991422 2861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:23.991448 2861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:23.992898 2861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:23.993093 2861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:23.993278 2861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:23.998723 2869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:23.998701 2866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:23.998696 2867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:23.998736 2861 server_base.cc:1034] running on GCE node
I20260620 08:04:23.999217 2861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:23.999447 2861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:24.000763 2861 hybrid_clock.cc:648] HybridClock initialized: now 1781942664000747 us; error 32 us; skew 500 ppm
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:24.003649 2861 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:24.004156 2861 webserver.cc:460] Webserver started at http://127.31.115.193:45521/ using document root <none> and password file <none>
I20260620 08:04:24.004423 2861 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:24.004505 2861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:24.004662 2861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:24.005854 2861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "769bfe2fcb9d419fb12ecdbd66a91a6a"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "19da6e0b57ce7781f236f720fa69bbf0"
server_key_iv: "e92e1f6b9c9511b099081f1f926921ed"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.006237 2861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "769bfe2fcb9d419fb12ecdbd66a91a6a"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "19da6e0b57ce7781f236f720fa69bbf0"
server_key_iv: "e92e1f6b9c9511b099081f1f926921ed"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.008044 2861 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:24.008890 2876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.009191 2861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:24.009317 2861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "769bfe2fcb9d419fb12ecdbd66a91a6a"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "19da6e0b57ce7781f236f720fa69bbf0"
server_key_iv: "e92e1f6b9c9511b099081f1f926921ed"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.009418 2861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:24.018507 2861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:24.021178 2861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:24.021354 2861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:24.021685 2861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:24.022030 2861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:24.022086 2861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.022132 2861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:24.022158 2861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.028393 2861 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:39533
I20260620 08:04:24.028439 2989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:39533 every 8 connection(s)
I20260620 08:04:24.028781 2861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:24.030537 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2861
I20260620 08:04:24.030656 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:24.030922 32207 external_mini_cluster.cc:1428] Setting key 33f044217de45dabd81cdd0ad04391da
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:04:24.040529 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.029692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:49147 (local address 127.31.115.254:38121)
0620 08:04:24.029843 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:24.029845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:24.029909 (+ 64us) server_negotiation.cc:407] Connection header received
0620 08:04:24.030467 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:24.030469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:24.030486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:24.030581 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:24.031799 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.032631 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.034422 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.034623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.034685 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.037336 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:24.037338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:24.037338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:24.037345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:24.038950 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.039452 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.039453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.039454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.039520 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.039817 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.039818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.039818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.039948 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:24.040023 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:24.040328 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 08:04:24.040420 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 08:04:24.040735 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.029529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:38121 (local address 127.31.115.193:49147)
0620 08:04:24.029780 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:24.029840 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:24.030343 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:24.030795 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:24.030805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:24.031480 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:24.031703 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:24.031709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.032821 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.032822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:24.034330 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:24.034333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.034717 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.034718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:24.034878 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.035423 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:24.035434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:24.037254 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:24.039072 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:24.039081 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:24.039083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:24.039350 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:24.039593 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:24.039594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:24.039594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:24.039734 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:24.040028 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:24.040036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:24.040190 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 08:04:24.040306 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 08:04:24.040477 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":124,"threads_started":1}
I20260620 08:04:24.041265 2990 heartbeater.cc:347] Connected to a master server at 127.31.115.254:38121
I20260620 08:04:24.041360 2990 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:24.041597 2990 heartbeater.cc:511] Master 127.31.115.254:38121 requested a full tablet report, sending...
I20260620 08:04:24.042222 2799 ts_manager.cc:194] Registered new tserver with Master: 769bfe2fcb9d419fb12ecdbd66a91a6a (127.31.115.193:39533)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
I20260620 08:04:24.043053 2799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:49147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:24.057070 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38121
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:24.138440 2996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:24.138695 2996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:24.138736 2996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:24.139991 2996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:24.140064 2996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:24.140107 2996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:24.141592 2996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:24.141906 2996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:24.142148 2996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:24.147154 3002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:24.147228 3004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:24.147300 2996 server_base.cc:1034] running on GCE node
W20260620 08:04:24.147156 3001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:24.147651 2996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:24.147956 2996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:24.149180 2996 hybrid_clock.cc:648] HybridClock initialized: now 1781942664149159 us; error 41 us; skew 500 ppm
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:24.152132 2996 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:24.152695 2996 webserver.cc:460] Webserver started at http://127.31.115.194:42597/ using document root <none> and password file <none>
I20260620 08:04:24.152936 2996 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:24.153005 2996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:24.153168 2996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:24.154327 2996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "1c8e3b7c39a24699bcb184ad3bb45616"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "e3f6d14ca61f70f16c8b7888b939e0e5"
server_key_iv: "a90209cd59038bc38df30e0511dbf384"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.154704 2996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "1c8e3b7c39a24699bcb184ad3bb45616"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "e3f6d14ca61f70f16c8b7888b939e0e5"
server_key_iv: "a90209cd59038bc38df30e0511dbf384"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.156072 2996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:04:24.157047 3011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.157269 2996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:24.157383 2996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "1c8e3b7c39a24699bcb184ad3bb45616"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "e3f6d14ca61f70f16c8b7888b939e0e5"
server_key_iv: "a90209cd59038bc38df30e0511dbf384"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.157460 2996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:24.164757 2996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:24.167064 2996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:24.167227 2996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:24.167454 2996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:24.167776 2996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:24.167835 2996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.167877 2996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:24.167906 2996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.174278 2996 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:43007
I20260620 08:04:24.174321 3124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:43007 every 8 connection(s)
I20260620 08:04:24.174664 2996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:24.182420 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 2996
I20260620 08:04:24.182549 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:24.182757 32207 external_mini_cluster.cc:1428] Setting key c9dcfb668c355adb46a152a29313cacf
I20260620 08:04:24.185441 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.175419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:40339 (local address 127.31.115.254:38121)
0620 08:04:24.175515 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:24.175518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:24.175685 (+ 167us) server_negotiation.cc:407] Connection header received
0620 08:04:24.176174 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:24.176175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:24.176186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:24.176266 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:24.177390 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.178110 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.179297 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.179516 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.179580 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.182138 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:24.182140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:24.182141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:24.182152 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:24.183863 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.184284 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.184286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.184287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.184352 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.184618 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.184619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.184619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.184799 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:24.184878 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:24.185208 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 08:04:24.185317 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:04:24.185483 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.175377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:38121 (local address 127.31.115.194:40339)
0620 08:04:24.175561 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:24.175599 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:24.176076 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:24.176341 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:24.176344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:24.177050 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:24.177275 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:24.177280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.178222 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.178223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:24.179198 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:24.179200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.179614 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.179616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:24.179771 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.180332 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:24.180342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:24.182005 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:24.183949 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:24.183958 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:24.183961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:24.184202 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:24.184426 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:24.184427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:24.184427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:24.184546 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:24.184901 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:24.184909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:24.185061 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 08:04:24.185163 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:04:24.185301 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260620 08:04:24.186158 3125 heartbeater.cc:347] Connected to a master server at 127.31.115.254:38121
I20260620 08:04:24.186306 3125 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:24.186540 3125 heartbeater.cc:511] Master 127.31.115.254:38121 requested a full tablet report, sending...
I20260620 08:04:24.187054 2799 ts_manager.cc:194] Registered new tserver with Master: 1c8e3b7c39a24699bcb184ad3bb45616 (127.31.115.194:43007)
I20260620 08:04:24.187469 2799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:40339
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:24.208954 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38121
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:24.294140 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:24.294356 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:24.294382 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:24.295509 3130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:24.295547 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:24.295573 3130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:24.296980 3130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:24.297181 3130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:24.297361 3130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:24.302691 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:24.302699 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:24.302786 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:24.303457 3130 server_base.cc:1034] running on GCE node
I20260620 08:04:24.303753 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:24.304013 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:24.305178 3130 hybrid_clock.cc:648] HybridClock initialized: now 1781942664305157 us; error 37 us; skew 500 ppm
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:24.307940 3130 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:24.308408 3130 webserver.cc:460] Webserver started at http://127.31.115.195:37865/ using document root <none> and password file <none>
I20260620 08:04:24.308666 3130 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:24.308749 3130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:24.308902 3130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:24.310065 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "7cdcd61abbef40a58ca8dfc493ed0533"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "880b248b8fc908d902661f7d512880c1"
server_key_iv: "56a8f167bb3841b5091c0b5330f5177a"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.310426 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "7cdcd61abbef40a58ca8dfc493ed0533"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "880b248b8fc908d902661f7d512880c1"
server_key_iv: "56a8f167bb3841b5091c0b5330f5177a"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.311847 3130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:24.312750 3145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.313030 3130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:24.313134 3130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "7cdcd61abbef40a58ca8dfc493ed0533"
format_stamp: "Formatted at 2026-06-20 08:04:24 on dist-test-slave-c4w8"
server_key: "880b248b8fc908d902661f7d512880c1"
server_key_iv: "56a8f167bb3841b5091c0b5330f5177a"
server_key_version: "encryptionkey@0"
I20260620 08:04:24.313226 3130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:24.327302 3130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:24.329835 3130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:24.330006 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:24.330263 3130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:24.330587 3130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:24.330641 3130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.330690 3130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:24.330737 3130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:24.336707 3130 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:42233
I20260620 08:04:24.336737 3258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:42233 every 8 connection(s)
I20260620 08:04:24.337122 3130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:24.344565 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3130
I20260620 08:04:24.344688 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:24.344926 32207 external_mini_cluster.cc:1428] Setting key a2210ea1a5e322f3284c35577b02aaeb
I20260620 08:04:24.347044 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.337880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:35713 (local address 127.31.115.254:38121)
0620 08:04:24.337958 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:24.337960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:24.338241 (+ 281us) server_negotiation.cc:407] Connection header received
0620 08:04:24.338778 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:24.338780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:24.338791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:24.338846 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:24.339840 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.340355 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.341368 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.341554 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.341661 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.343921 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:24.343923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:24.343923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:24.343930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:24.345638 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.346144 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.346145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.346145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.346217 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.346498 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.346499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.346499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.346606 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:24.346657 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:24.346914 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 08:04:24.346979 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:04:24.347355 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.337875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:38121 (local address 127.31.115.195:35713)
0620 08:04:24.338069 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:24.338133 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:24.338642 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:24.338926 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:24.338945 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:24.339549 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:24.339747 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:24.339752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.340441 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.340442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:24.341283 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:24.341285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.341681 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.341682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:24.341804 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.342299 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:24.342308 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:24.343829 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:24.345705 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:24.345713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:24.345715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:24.346076 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:24.346306 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:24.346307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:24.346307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:24.346430 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:24.346649 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:24.346653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:24.346794 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 08:04:24.346888 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 08:04:24.347057 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260620 08:04:24.347934 3259 heartbeater.cc:347] Connected to a master server at 127.31.115.254:38121
I20260620 08:04:24.348058 3259 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:24.348279 3259 heartbeater.cc:511] Master 127.31.115.254:38121 requested a full tablet report, sending...
I20260620 08:04:24.348726 2799 ts_manager.cc:194] Registered new tserver with Master: 7cdcd61abbef40a58ca8dfc493ed0533 (127.31.115.195:42233)
I20260620 08:04:24.349110 2799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:35713
I20260620 08:04:24.357023 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942664, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:24.382107 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:24.373778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.31.115.254:38121)
0620 08:04:24.373838 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:24.373840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:24.374029 (+ 189us) server_negotiation.cc:407] Connection header received
0620 08:04:24.374071 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:24.374072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:24.374084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:24.374133 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:24.374774 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.375335 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.376162 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:24.376367 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:24.376432 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:24.379241 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:24.379243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:24.379243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:24.379252 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:24.380800 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.381199 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.381201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.381201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.381261 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:24.381500 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:24.381500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:24.381501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:24.381761 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:24.381859 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:24.381947 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 08:04:24.382015 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 08:04:24.382380 2799 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43592
I20260620 08:04:24.382839 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2861
I20260620 08:04:24.387655 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2996
I20260620 08:04:24.391875 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3130
I20260620 08:04:24.396241 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 2767
2026-06-20T08:04:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2743 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[3271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[3271](info): set up 2 sockets
Jun 20 08:04:24 dist-test-slave-c4w8 krb5kdc[3271](info): commencing operation
krb5kdc: starting...
W20260620 08:04:25.466409 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:25 dist-test-slave-c4w8 krb5kdc[3271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:25.589049 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:37481
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:37481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:25.671797 3284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:25.672127 3284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:25.672173 3284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:25.673959 3284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:25.674062 3284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:25.674091 3284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:25.674116 3284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:25.674149 3284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:25.676192 3284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:37481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:25.676523 3284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:25.676787 3284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:25.682156 3292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:25.682161 3284 server_base.cc:1034] running on GCE node
W20260620 08:04:25.682078 3290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:25.682168 3289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:25.682607 3284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:25.682916 3284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:25.684087 3284 hybrid_clock.cc:648] HybridClock initialized: now 1781942665684078 us; error 37 us; skew 500 ppm
Jun 20 08:04:25 dist-test-slave-c4w8 krb5kdc[3271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942665, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:25.686739 3284 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:25.687232 3284 webserver.cc:460] Webserver started at http://127.31.115.254:33883/ using document root <none> and password file <none>
I20260620 08:04:25.687516 3284 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:25.687573 3284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:25.687708 3284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:25.688840 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "3b473cf9799143e38b6e54014031c2c2"
format_stamp: "Formatted at 2026-06-20 08:04:25 on dist-test-slave-c4w8"
server_key: "29d340eab9210e7c4205ae7abf263a83"
server_key_iv: "6f41c65792b15005dda3ea638771b9e5"
server_key_version: "encryptionkey@0"
I20260620 08:04:25.689211 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "3b473cf9799143e38b6e54014031c2c2"
format_stamp: "Formatted at 2026-06-20 08:04:25 on dist-test-slave-c4w8"
server_key: "29d340eab9210e7c4205ae7abf263a83"
server_key_iv: "6f41c65792b15005dda3ea638771b9e5"
server_key_version: "encryptionkey@0"
I20260620 08:04:25.690721 3284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:25.691529 3299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:25.691789 3284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:25.691898 3284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "3b473cf9799143e38b6e54014031c2c2"
format_stamp: "Formatted at 2026-06-20 08:04:25 on dist-test-slave-c4w8"
server_key: "29d340eab9210e7c4205ae7abf263a83"
server_key_iv: "6f41c65792b15005dda3ea638771b9e5"
server_key_version: "encryptionkey@0"
I20260620 08:04:25.691982 3284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:25.700526 3284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:25.702919 3284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:25.703094 3284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:25.707620 3284 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:37481
I20260620 08:04:25.707655 3351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:37481 every 8 connection(s)
I20260620 08:04:25.708171 3284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:25.708912 3352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:25.711511 3352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2: Bootstrap starting.
I20260620 08:04:25.712293 3352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:25.712630 3352 log.cc:824] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:25.713582 3352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2: No bootstrap required, opened a new log
I20260620 08:04:25.714257 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3284
I20260620 08:04:25.714341 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:25.714533 32207 external_mini_cluster.cc:1428] Setting key 03f96ac0930b2456682f8450950c10a9
I20260620 08:04:25.716176 3352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b473cf9799143e38b6e54014031c2c2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37481 } }
I20260620 08:04:25.716367 3352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:25.716414 3352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b473cf9799143e38b6e54014031c2c2, State: Initialized, Role: FOLLOWER
I20260620 08:04:25.716598 3352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b473cf9799143e38b6e54014031c2c2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37481 } }
I20260620 08:04:25.716699 3352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:25.716735 3352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:25.716804 3352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:25.717562 3352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b473cf9799143e38b6e54014031c2c2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37481 } }
I20260620 08:04:25.717726 3352 leader_election.cc:302] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b473cf9799143e38b6e54014031c2c2; no voters:
I20260620 08:04:25.717926 3352 leader_election.cc:288] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:25.717984 3357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:25.718174 3357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3b473cf9799143e38b6e54014031c2c2, State: Running, Role: LEADER
I20260620 08:04:25.718205 3352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:25.718288 3357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b473cf9799143e38b6e54014031c2c2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37481 } }
I20260620 08:04:25.718626 3357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b473cf9799143e38b6e54014031c2c2. Latest consensus state: current_term: 1 leader_uuid: "3b473cf9799143e38b6e54014031c2c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b473cf9799143e38b6e54014031c2c2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37481 } } }
I20260620 08:04:25.718644 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b473cf9799143e38b6e54014031c2c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b473cf9799143e38b6e54014031c2c2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37481 } } }
I20260620 08:04:25.718876 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:25.719161 3366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:25.718773 3357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:25.719878 3366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:25.721957 3366 catalog_manager.cc:1347] Generated new cluster ID: def67de6651549df9fa1db57861a600d
I20260620 08:04:25.722016 3366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:25 dist-test-slave-c4w8 krb5kdc[3271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:25.727308 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:25.715112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.31.115.254:37481)
0620 08:04:25.715330 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:25.715347 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:25.715372 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:04:25.716235 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:25.716247 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:25.716324 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:25.716547 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:25.717378 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:25.718812 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:25.719904 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:25.720525 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:25.720632 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:25.723453 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:25.723458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:25.723458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:25.723471 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:25.725526 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:25.725947 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:25.725950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:25.725951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:25.726078 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:25.726288 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:25.726289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:25.726289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:25.726652 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:25.726870 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:25.726949 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:04:25.727100 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260620 08:04:25.746994 3366 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:25.747761 3366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:25.753262 3366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b473cf9799143e38b6e54014031c2c2: Generated new TSK 0
I20260620 08:04:25.753463 3366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:25.784634 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37481
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:25.869046 3378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:25.869314 3378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:25.869359 3378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:25.870646 3378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:25.870724 3378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:25.870769 3378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:25.872316 3378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:25.872605 3378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:25.872859 3378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:25.878170 3384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:25.878311 3383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:25.878170 3386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:25.878597 3378 server_base.cc:1034] running on GCE node
I20260620 08:04:25.878852 3378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:25.879128 3378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:25.880327 3378 hybrid_clock.cc:648] HybridClock initialized: now 1781942665880307 us; error 29 us; skew 500 ppm
Jun 20 08:04:25 dist-test-slave-c4w8 krb5kdc[3271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942665, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:25.883150 3378 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:25.883637 3378 webserver.cc:460] Webserver started at http://127.31.115.193:43297/ using document root <none> and password file <none>
I20260620 08:04:25.883888 3378 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:25.883957 3378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:25.884125 3378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:25.885222 3378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "e227b49714f449f59ad67e0ea6639ab6"
format_stamp: "Formatted at 2026-06-20 08:04:25 on dist-test-slave-c4w8"
server_key: "e665789a5e7f6ec254b45d4b620db388"
server_key_iv: "63e0bdcdb3c6f27cd8baf4bf0468efd5"
server_key_version: "encryptionkey@0"
I20260620 08:04:25.885578 3378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "e227b49714f449f59ad67e0ea6639ab6"
format_stamp: "Formatted at 2026-06-20 08:04:25 on dist-test-slave-c4w8"
server_key: "e665789a5e7f6ec254b45d4b620db388"
server_key_iv: "63e0bdcdb3c6f27cd8baf4bf0468efd5"
server_key_version: "encryptionkey@0"
I20260620 08:04:25.887040 3378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:25.888012 3393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:25.888374 3378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:25.888491 3378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "e227b49714f449f59ad67e0ea6639ab6"
format_stamp: "Formatted at 2026-06-20 08:04:25 on dist-test-slave-c4w8"
server_key: "e665789a5e7f6ec254b45d4b620db388"
server_key_iv: "63e0bdcdb3c6f27cd8baf4bf0468efd5"
server_key_version: "encryptionkey@0"
I20260620 08:04:25.888625 3378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:25.911456 3378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:25.914029 3378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:25.914209 3378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:25.914580 3378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:25.914922 3378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:25.914984 3378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:25.915032 3378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:25.915055 3378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:25.922021 3378 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:40087
I20260620 08:04:25.922073 3506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:40087 every 8 connection(s)
I20260620 08:04:25.922422 3378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:04:25 dist-test-slave-c4w8 krb5kdc[3271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942665, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:25.930573 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3378
I20260620 08:04:25.930689 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:25.930912 32207 external_mini_cluster.cc:1428] Setting key cc4f52b0745544e87e9e7761482799a2
I20260620 08:04:25.933539 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:25.923388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:36507 (local address 127.31.115.254:37481)
0620 08:04:25.923494 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:25.923497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:25.923671 (+ 174us) server_negotiation.cc:407] Connection header received
0620 08:04:25.924737 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:25.924739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:25.924753 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:25.924804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:25.925509 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:25.926102 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:25.927275 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:25.927517 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:25.927618 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:25.930085 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:25.930087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:25.930087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:25.930096 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:25.932036 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:25.932466 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:25.932467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:25.932467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:25.932535 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:25.932797 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:25.932798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:25.932798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:25.932956 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:25.933024 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:25.933349 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 08:04:25.933443 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 08:04:25.933791 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:25.923275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37481 (local address 127.31.115.193:36507)
0620 08:04:25.923523 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:25.923574 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:25.924593 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:25.924883 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:25.924886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:25.925164 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:25.925369 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:25.925373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:25.926229 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:25.926235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:25.927066 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:25.927068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:25.927635 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:25.927636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:25.927780 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:25.928408 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:25.928418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:25.929959 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:25.932121 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:25.932129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:25.932131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:25.932388 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:25.932611 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:25.932611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:25.932611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:25.932730 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:25.933051 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:25.933062 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:25.933197 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 08:04:25.933323 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 08:04:25.933504 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260620 08:04:25.934470 3507 heartbeater.cc:347] Connected to a master server at 127.31.115.254:37481
I20260620 08:04:25.934583 3507 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:25.934803 3507 heartbeater.cc:511] Master 127.31.115.254:37481 requested a full tablet report, sending...
I20260620 08:04:25.935442 3316 ts_manager.cc:194] Registered new tserver with Master: e227b49714f449f59ad67e0ea6639ab6 (127.31.115.193:40087)
I20260620 08:04:25.936124 3316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:36507
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:25.957129 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37481
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:26.044171 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:26.044431 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:26.044469 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:26.045708 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:26.045783 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:26.045830 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:26.047362 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:26.047637 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:26.047871 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:26.053107 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:26.053068 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:26.053047 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:26.053330 3512 server_base.cc:1034] running on GCE node
I20260620 08:04:26.053565 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:26.053864 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:26.055058 3512 hybrid_clock.cc:648] HybridClock initialized: now 1781942666055025 us; error 47 us; skew 500 ppm
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:26.057858 3512 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:26.058370 3512 webserver.cc:460] Webserver started at http://127.31.115.194:46095/ using document root <none> and password file <none>
I20260620 08:04:26.058619 3512 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:26.058691 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:26.058847 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:26.059943 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "9c7e4244aee14b07a7f2a8d6b9d7b00c"
format_stamp: "Formatted at 2026-06-20 08:04:26 on dist-test-slave-c4w8"
server_key: "79c28e06d7981f3e8833885370cccb81"
server_key_iv: "4ac7ae97bdc6677dabb643f472b61134"
server_key_version: "encryptionkey@0"
I20260620 08:04:26.060320 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "9c7e4244aee14b07a7f2a8d6b9d7b00c"
format_stamp: "Formatted at 2026-06-20 08:04:26 on dist-test-slave-c4w8"
server_key: "79c28e06d7981f3e8833885370cccb81"
server_key_iv: "4ac7ae97bdc6677dabb643f472b61134"
server_key_version: "encryptionkey@0"
I20260620 08:04:26.061887 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:26.062772 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:26.063004 3512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:26.063117 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "9c7e4244aee14b07a7f2a8d6b9d7b00c"
format_stamp: "Formatted at 2026-06-20 08:04:26 on dist-test-slave-c4w8"
server_key: "79c28e06d7981f3e8833885370cccb81"
server_key_iv: "4ac7ae97bdc6677dabb643f472b61134"
server_key_version: "encryptionkey@0"
I20260620 08:04:26.063216 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:26.080546 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:26.083032 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:26.083204 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:26.083490 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:26.083823 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:26.083878 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:26.083921 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:26.083950 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:26.090186 3512 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:35429
I20260620 08:04:26.090281 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:35429 every 8 connection(s)
I20260620 08:04:26.090629 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:26.092350 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3512
I20260620 08:04:26.092478 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:26.092685 32207 external_mini_cluster.cc:1428] Setting key 53e8a42cfdb23514a219a2795ae6e1ab
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:04:26.102408 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:26.091605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:44313 (local address 127.31.115.254:37481)
0620 08:04:26.091681 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:26.091684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:26.091847 (+ 163us) server_negotiation.cc:407] Connection header received
0620 08:04:26.092942 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:26.092944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:26.092958 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:26.093002 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:26.093817 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.094606 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.095990 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.096187 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.096247 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:26.098851 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:26.098853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:26.098853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:26.098863 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:26.100676 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.101160 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.101162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.101162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.101227 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.101489 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.101490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.101490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.101694 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:26.101763 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:26.102223 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 08:04:26.102323 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:26.102582 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:26.091542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37481 (local address 127.31.115.194:44313)
0620 08:04:26.091714 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:26.091758 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:26.092847 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:26.093084 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:26.093088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:26.093456 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:26.093728 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:26.093733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.094773 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.094774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:26.095920 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:26.095923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.096287 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.096288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:26.096450 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:26.096987 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:26.096998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:26.098744 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:26.100812 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:26.100819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:26.100822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:26.101073 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:26.101308 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:26.101309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:26.101309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:26.101417 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:26.101758 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:26.101766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:26.101983 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 08:04:26.102176 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 08:04:26.102338 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260620 08:04:26.103127 3641 heartbeater.cc:347] Connected to a master server at 127.31.115.254:37481
I20260620 08:04:26.103240 3641 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:26.103461 3641 heartbeater.cc:511] Master 127.31.115.254:37481 requested a full tablet report, sending...
I20260620 08:04:26.103964 3316 ts_manager.cc:194] Registered new tserver with Master: 9c7e4244aee14b07a7f2a8d6b9d7b00c (127.31.115.194:35429)
I20260620 08:04:26.104409 3316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:44313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:26.122364 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37481
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:26.206017 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:26.206235 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:26.206260 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:26.207459 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:26.207507 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:26.207536 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:26.209317 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:26.209599 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:26.209899 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:26.215600 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:26.215601 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:26.215600 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:26.216293 3646 server_base.cc:1034] running on GCE node
I20260620 08:04:26.216503 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:26.216802 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:26.218011 3646 hybrid_clock.cc:648] HybridClock initialized: now 1781942666217989 us; error 43 us; skew 500 ppm
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:26.221045 3646 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:26.221515 3646 webserver.cc:460] Webserver started at http://127.31.115.195:37705/ using document root <none> and password file <none>
I20260620 08:04:26.221776 3646 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:26.221823 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:26.221979 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:26.223062 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "09f44c1611f94e68ab1bcb0eb633c104"
format_stamp: "Formatted at 2026-06-20 08:04:26 on dist-test-slave-c4w8"
server_key: "e54e0394a8043ef02d175e65736322ae"
server_key_iv: "e5c495dda74f4474b5d10133eeae9ba5"
server_key_version: "encryptionkey@0"
I20260620 08:04:26.223457 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "09f44c1611f94e68ab1bcb0eb633c104"
format_stamp: "Formatted at 2026-06-20 08:04:26 on dist-test-slave-c4w8"
server_key: "e54e0394a8043ef02d175e65736322ae"
server_key_iv: "e5c495dda74f4474b5d10133eeae9ba5"
server_key_version: "encryptionkey@0"
I20260620 08:04:26.224912 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:26.225889 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:26.226137 3646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:26.226230 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "09f44c1611f94e68ab1bcb0eb633c104"
format_stamp: "Formatted at 2026-06-20 08:04:26 on dist-test-slave-c4w8"
server_key: "e54e0394a8043ef02d175e65736322ae"
server_key_iv: "e5c495dda74f4474b5d10133eeae9ba5"
server_key_version: "encryptionkey@0"
I20260620 08:04:26.226334 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:26.237560 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:26.240139 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:26.240302 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:26.240573 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:26.240918 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:26.240952 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:26.240975 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:26.241024 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:26.247087 3646 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:34519
I20260620 08:04:26.247182 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:34519 every 8 connection(s)
I20260620 08:04:26.247511 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:26.257824 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3646
I20260620 08:04:26.257931 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:26.257993 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:26.248479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:45205 (local address 127.31.115.254:37481)
0620 08:04:26.248563 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:26.248566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:26.248700 (+ 134us) server_negotiation.cc:407] Connection header received
0620 08:04:26.249710 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:26.249712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:26.249734 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:26.249790 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:26.250438 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.250953 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.252023 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.252227 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.252330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:26.254885 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:26.254886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:26.254886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:26.254894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:26.256365 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.256815 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.256816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.256816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.256878 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.257192 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.257193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.257193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.257310 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:26.257366 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:26.257784 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 08:04:26.257873 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:04:26.258157 32207 external_mini_cluster.cc:1428] Setting key cf6429be822e14da073d744f59490884
I20260620 08:04:26.258205 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:26.248395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37481 (local address 127.31.115.195:45205)
0620 08:04:26.248560 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:26.248612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:26.249592 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:26.249849 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:26.249853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:26.250140 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:26.250348 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:26.250351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.251073 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.251074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:26.251936 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:26.251938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.252336 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.252338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:26.252503 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:26.253199 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:26.253212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:26.254764 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:26.256454 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:26.256460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:26.256462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:26.256715 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:26.256966 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:26.256967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:26.256967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:26.257085 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:26.257413 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:26.257419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:26.257553 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 08:04:26.257737 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 08:04:26.257923 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260620 08:04:26.258888 3775 heartbeater.cc:347] Connected to a master server at 127.31.115.254:37481
I20260620 08:04:26.259016 3775 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:26.259217 3775 heartbeater.cc:511] Master 127.31.115.254:37481 requested a full tablet report, sending...
I20260620 08:04:26.259613 3315 ts_manager.cc:194] Registered new tserver with Master: 09f44c1611f94e68ab1bcb0eb633c104 (127.31.115.195:34519)
I20260620 08:04:26.259990 3315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:45205
I20260620 08:04:26.270741 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:26.296411 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:26.287138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.31.115.193:40087)
0620 08:04:26.287350 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:26.287355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:26.287368 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:04:26.287395 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:26.287397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:26.287412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:26.287492 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:26.288157 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.288973 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.289814 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.290172 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.290234 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:26.292902 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:26.292910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:26.292911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:26.292918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:26.294798 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.295252 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.295256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.295256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.295334 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.295615 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.295617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.295617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.295779 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:26.295904 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:26.296130 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 08:04:26.296216 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:26.305276 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:26.297908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.31.115.254:37481)
0620 08:04:26.297989 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:26.297992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:26.298002 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:26.298079 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:26.298080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:26.298090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:26.298154 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:26.298665 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.299158 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.299840 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:26.300038 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:26.300115 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:26.302535 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:26.302537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:26.302537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:26.302545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:26.304028 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.304411 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.304412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.304413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.304469 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:26.304733 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:26.304734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:26.304735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:26.304891 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:26.304973 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:26.305121 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 08:04:26.305209 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 08:04:26.305577 3315 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48682
I20260620 08:04:26.306257 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3378
I20260620 08:04:26.311414 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3512
I20260620 08:04:26.317121 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3646
I20260620 08:04:26.321929 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3284
2026-06-20T08:04:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1926 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3787](info): set up 2 sockets
Jun 20 08:04:26 dist-test-slave-c4w8 krb5kdc[3787](info): commencing operation
krb5kdc: starting...
W20260620 08:04:27.394121 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:27 dist-test-slave-c4w8 krb5kdc[3787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:27.521222 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33543
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:27.609815 3800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:27.610087 3800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:27.610132 3800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:27.611327 3800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:27.611413 3800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:27.611442 3800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:27.611482 3800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:27.611539 3800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:27.613142 3800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:33543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:27.613453 3800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:27.613739 3800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:27.619019 3806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:27.619062 3805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:27.619029 3808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:27.619374 3800 server_base.cc:1034] running on GCE node
I20260620 08:04:27.619618 3800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:27.619926 3800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:27.621096 3800 hybrid_clock.cc:648] HybridClock initialized: now 1781942667621063 us; error 49 us; skew 500 ppm
Jun 20 08:04:27 dist-test-slave-c4w8 krb5kdc[3787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942667, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:27.626888 3800 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:27.627285 3800 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:27.627336 3800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:27.627429 3800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:27.628463 3800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "f89e3e1597604fbf8ddc320053e6f4d9"
format_stamp: "Formatted at 2026-06-20 08:04:27 on dist-test-slave-c4w8"
server_key: "95ff1ac4b3b7bd932c64032d4b16e2f6"
server_key_iv: "c0029063a01e77e234c086d846620fdc"
server_key_version: "encryptionkey@0"
I20260620 08:04:27.628778 3800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "f89e3e1597604fbf8ddc320053e6f4d9"
format_stamp: "Formatted at 2026-06-20 08:04:27 on dist-test-slave-c4w8"
server_key: "95ff1ac4b3b7bd932c64032d4b16e2f6"
server_key_iv: "c0029063a01e77e234c086d846620fdc"
server_key_version: "encryptionkey@0"
I20260620 08:04:27.630121 3800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:27.630921 3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:27.631130 3800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:27.631227 3800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "f89e3e1597604fbf8ddc320053e6f4d9"
format_stamp: "Formatted at 2026-06-20 08:04:27 on dist-test-slave-c4w8"
server_key: "95ff1ac4b3b7bd932c64032d4b16e2f6"
server_key_iv: "c0029063a01e77e234c086d846620fdc"
server_key_version: "encryptionkey@0"
I20260620 08:04:27.631284 3800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:27.644685 3800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:27.647377 3800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:27.647500 3800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:27.651741 3800 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:33543
I20260620 08:04:27.651856 3866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:33543 every 8 connection(s)
I20260620 08:04:27.652132 3800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:27.653204 3867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:27.655748 3867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9: Bootstrap starting.
I20260620 08:04:27.656428 3867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:27.656762 3867 log.cc:824] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:27.656791 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3800
I20260620 08:04:27.656900 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:27.657110 32207 external_mini_cluster.cc:1428] Setting key bfd530ee999d97b9064e2907613cc8dc
I20260620 08:04:27.657605 3867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9: No bootstrap required, opened a new log
I20260620 08:04:27.659370 3867 raft_consensus.cc:348] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33543 } }
I20260620 08:04:27.659541 3867 raft_consensus.cc:374] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:27.659590 3867 raft_consensus.cc:729] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89e3e1597604fbf8ddc320053e6f4d9, State: Initialized, Role: FOLLOWER
I20260620 08:04:27.659731 3867 consensus_queue.cc:260] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33543 } }
I20260620 08:04:27.659824 3867 raft_consensus.cc:388] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:27.659860 3867 raft_consensus.cc:482] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:27.659902 3867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:27.660908 3867 raft_consensus.cc:504] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33543 } }
I20260620 08:04:27.661003 3867 leader_election.cc:302] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f89e3e1597604fbf8ddc320053e6f4d9; no voters:
I20260620 08:04:27.661228 3867 leader_election.cc:288] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:27.661259 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:27.661367 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [term 1 LEADER]: Becoming Leader. State: Replica: f89e3e1597604fbf8ddc320053e6f4d9, State: Running, Role: LEADER
I20260620 08:04:27.661461 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33543 } }
I20260620 08:04:27.661504 3867 sys_catalog.cc:564] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:27.661859 3872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f89e3e1597604fbf8ddc320053e6f4d9. Latest consensus state: current_term: 1 leader_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33543 } } }
I20260620 08:04:27.661882 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89e3e1597604fbf8ddc320053e6f4d9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33543 } } }
I20260620 08:04:27.661935 3872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:27.661965 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:27.662338 3878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:27.662925 3878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:27.664542 3878 catalog_manager.cc:1347] Generated new cluster ID: 5b38e560bf3945cfa5077fbba7b269fe
I20260620 08:04:27.664597 3878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:27 dist-test-slave-c4w8 krb5kdc[3787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:27.670855 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:27.657751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.31.115.254:33543)
0620 08:04:27.657965 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:27.657974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:27.658000 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:04:27.659359 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:27.659368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:27.659428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:27.659604 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:27.660420 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:27.661573 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:27.662741 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:27.663527 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:27.663653 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:27.666495 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:27.666499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:27.666500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:27.666514 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:27.668811 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:27.669248 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:27.669252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:27.669253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:27.669409 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:27.669662 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:27.669663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:27.669664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:27.670050 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:27.670242 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:27.670335 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 08:04:27.670538 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
I20260620 08:04:27.680804 3878 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:27.681756 3878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:27.693532 3878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9: Generated new TSK 0
I20260620 08:04:27.693820 3878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:27.725914 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33543
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:27.811681 3893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:27.811939 3893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:27.811983 3893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:27.813190 3893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:27.813262 3893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:27.813310 3893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:27.814922 3893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:27.815213 3893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:27.815465 3893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:27.820746 3899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:27.820649 3898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:27.820976 3893 server_base.cc:1034] running on GCE node
W20260620 08:04:27.820652 3901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:27.821391 3893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:27.821646 3893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:27.822889 3893 hybrid_clock.cc:648] HybridClock initialized: now 1781942667822869 us; error 39 us; skew 500 ppm
Jun 20 08:04:27 dist-test-slave-c4w8 krb5kdc[3787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942667, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:27.825904 3893 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:27.826309 3893 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:27.826364 3893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:27.826467 3893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:27.827607 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "ba808e99fd134405a36e5e61ed483a09"
format_stamp: "Formatted at 2026-06-20 08:04:27 on dist-test-slave-c4w8"
server_key: "66d664bc6ff9ee116612f56559558cf9"
server_key_iv: "689f4f29726eee2595b0e32c4ba57ba5"
server_key_version: "encryptionkey@0"
I20260620 08:04:27.827944 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "ba808e99fd134405a36e5e61ed483a09"
format_stamp: "Formatted at 2026-06-20 08:04:27 on dist-test-slave-c4w8"
server_key: "66d664bc6ff9ee116612f56559558cf9"
server_key_iv: "689f4f29726eee2595b0e32c4ba57ba5"
server_key_version: "encryptionkey@0"
I20260620 08:04:27.829308 3893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:27.830289 3907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:27.830482 3893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:04:27.830580 3893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "ba808e99fd134405a36e5e61ed483a09"
format_stamp: "Formatted at 2026-06-20 08:04:27 on dist-test-slave-c4w8"
server_key: "66d664bc6ff9ee116612f56559558cf9"
server_key_iv: "689f4f29726eee2595b0e32c4ba57ba5"
server_key_version: "encryptionkey@0"
I20260620 08:04:27.830654 3893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:27.862251 3893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:27.864955 3893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:27.865137 3893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:27.865432 3893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:27.865936 3893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:27.866003 3893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:27.866050 3893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:27.866083 3893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:27.872803 3893 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:42493
I20260620 08:04:27.872843 4020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:42493 every 8 connection(s)
I20260620 08:04:27.873157 3893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:04:27 dist-test-slave-c4w8 krb5kdc[3787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942667, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:27.882272 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 3893
I20260620 08:04:27.882428 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:27.882674 32207 external_mini_cluster.cc:1428] Setting key 4cfc4e9645d3c43b4c38df4f737fa6d3
I20260620 08:04:27.884109 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:27.874060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:60655 (local address 127.31.115.254:33543)
0620 08:04:27.874253 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:27.874255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:27.874452 (+ 197us) server_negotiation.cc:407] Connection header received
0620 08:04:27.875049 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:27.875050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:27.875069 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:27.875130 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:27.876210 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:27.876782 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:27.877969 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:27.878220 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:27.878313 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:27.880699 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:27.880701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:27.880701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:27.880712 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:27.882211 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:27.882765 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:27.882767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:27.882768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:27.882836 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:27.883249 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:27.883250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:27.883250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:27.883389 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:27.883472 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:27.883953 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 08:04:27.884037 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:27.884421 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:27.873991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33543 (local address 127.31.115.193:60655)
0620 08:04:27.874314 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:27.874356 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:27.874925 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:27.875220 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:27.875224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:27.875510 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:27.876074 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:27.876080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:27.876910 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:27.876917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:27.877873 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:27.877876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:27.878322 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:27.878323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:27.878465 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:27.878969 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:27.878979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:27.880574 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:27.882321 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:27.882331 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:27.882333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:27.882677 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:27.882934 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:27.882935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:27.882935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:27.883111 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:27.883492 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:27.883507 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:27.883761 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 08:04:27.883900 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 08:04:27.884096 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20260620 08:04:27.885164 4021 heartbeater.cc:347] Connected to a master server at 127.31.115.254:33543
I20260620 08:04:27.885304 4021 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:27.885473 4021 heartbeater.cc:511] Master 127.31.115.254:33543 requested a full tablet report, sending...
I20260620 08:04:27.886117 3831 ts_manager.cc:194] Registered new tserver with Master: ba808e99fd134405a36e5e61ed483a09 (127.31.115.193:42493)
I20260620 08:04:27.887173 3831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:60655
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:27.909332 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33543
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:27.997100 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:27.997313 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:27.997337 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:27.998548 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:27.998596 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:27.998632 4026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:28.000241 4026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:28.000519 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:28.000771 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:28.006717 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.006732 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.006717 4026 server_base.cc:1034] running on GCE node
W20260620 08:04:28.006734 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.007161 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:28.007464 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:28.008740 4026 hybrid_clock.cc:648] HybridClock initialized: now 1781942668008692 us; error 58 us; skew 500 ppm
Jun 20 08:04:28 dist-test-slave-c4w8 krb5kdc[3787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942668, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:28.012138 4026 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:28.012836 4026 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:28.012912 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:28.013074 4026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:28.014894 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "93132fbbb2114f8b918bf1238a4d1e64"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "f532e333090a04d0782fc2808d47938c"
server_key_iv: "4568d48909b185291b1cb9e58ec268fd"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.015623 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "93132fbbb2114f8b918bf1238a4d1e64"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "f532e333090a04d0782fc2808d47938c"
server_key_iv: "4568d48909b185291b1cb9e58ec268fd"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.018138 4026 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 08:04:28.019176 4040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.019438 4026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:28.019605 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "93132fbbb2114f8b918bf1238a4d1e64"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "f532e333090a04d0782fc2808d47938c"
server_key_iv: "4568d48909b185291b1cb9e58ec268fd"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.019703 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:28.037461 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:28.040634 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:28.040824 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:28.041069 4026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:28.041393 4026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:28.041451 4026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.041491 4026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:28.041517 4026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.049017 4026 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:35503
I20260620 08:04:28.049045 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:35503 every 8 connection(s)
I20260620 08:04:28.049412 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:28.055315 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4026
I20260620 08:04:28.055469 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:28.055760 32207 external_mini_cluster.cc:1428] Setting key df18c91923202efa5205e8aaa76db9a6
Jun 20 08:04:28 dist-test-slave-c4w8 krb5kdc[3787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942668, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:28.062347 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.050514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:49023 (local address 127.31.115.254:33543)
0620 08:04:28.050582 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.050584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.050622 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:04:28.051444 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.051446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.051457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.051550 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:28.052570 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.053109 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.054199 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.054417 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.054514 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.057445 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:28.057447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:28.057447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:28.057457 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:28.060498 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:28.061006 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:28.061008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:28.061008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:28.061075 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:28.061477 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:28.061478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:28.061478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:28.061598 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:28.061714 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.062062 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.062168 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:04:28.062449 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.050286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33543 (local address 127.31.115.194:49023)
0620 08:04:28.050464 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.050518 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.051319 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.051691 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.051695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.051973 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:28.052459 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.052464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.053226 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.053227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.054110 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.054113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.054525 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.054527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.054652 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.055241 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:28.055253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:28.057333 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:28.060640 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:28.060649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:28.060652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:28.060920 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:28.061201 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:28.061202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:28.061202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:28.061374 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:28.061769 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:28.061776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:28.061913 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.062010 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.062142 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260620 08:04:28.063167 4154 heartbeater.cc:347] Connected to a master server at 127.31.115.254:33543
I20260620 08:04:28.063293 4154 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:28.063472 4154 heartbeater.cc:511] Master 127.31.115.254:33543 requested a full tablet report, sending...
I20260620 08:04:28.063916 3831 ts_manager.cc:194] Registered new tserver with Master: 93132fbbb2114f8b918bf1238a4d1e64 (127.31.115.194:35503)
I20260620 08:04:28.064303 3831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:49023
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:28.084558 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33543
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:28.169570 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:28.169857 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:28.169884 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:28.171106 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:28.171154 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:28.171207 4159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:28.172782 4159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:28.173043 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:28.173287 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:28.178742 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.178714 4167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.178869 4159 server_base.cc:1034] running on GCE node
W20260620 08:04:28.178766 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.179196 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:28.179502 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:28.180665 4159 hybrid_clock.cc:648] HybridClock initialized: now 1781942668180649 us; error 28 us; skew 500 ppm
Jun 20 08:04:28 dist-test-slave-c4w8 krb5kdc[3787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942668, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:28.183468 4159 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:28.183912 4159 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:28.183990 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:28.184161 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:28.185288 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "e5ccdc6b79d64e7e99dd260da24ee64a"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "eab0c4b49d356ad9f40521d452126567"
server_key_iv: "68ed7d2bf23ca6ee102d6cc88d11d43a"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.185745 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "e5ccdc6b79d64e7e99dd260da24ee64a"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "eab0c4b49d356ad9f40521d452126567"
server_key_iv: "68ed7d2bf23ca6ee102d6cc88d11d43a"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.187147 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:28.187989 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.188192 4159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:28.188297 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "e5ccdc6b79d64e7e99dd260da24ee64a"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "eab0c4b49d356ad9f40521d452126567"
server_key_iv: "68ed7d2bf23ca6ee102d6cc88d11d43a"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.188385 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:28.201242 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:28.203764 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:28.203938 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:28.204201 4159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:28.204530 4159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:28.204583 4159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.204627 4159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:28.204671 4159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.210603 4159 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:38359
I20260620 08:04:28.210642 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:38359 every 8 connection(s)
I20260620 08:04:28.210942 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:04:28 dist-test-slave-c4w8 krb5kdc[3787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942668, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:28.220084 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4159
I20260620 08:04:28.220208 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:28.220394 32207 external_mini_cluster.cc:1428] Setting key c09aee9eb71f40f3de2f0bfe78384f4d
I20260620 08:04:28.220928 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.211807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:57917 (local address 127.31.115.254:33543)
0620 08:04:28.211871 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.211873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.212100 (+ 227us) server_negotiation.cc:407] Connection header received
0620 08:04:28.212637 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.212638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.212653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.212701 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:28.213703 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.214216 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.215226 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.215443 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.215518 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.217899 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:28.217900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:28.217901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:28.217909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:28.219439 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:28.219930 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:28.219931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:28.219932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:28.219982 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:28.220238 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:28.220239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:28.220240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:28.220375 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:28.220447 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.220747 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.220830 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:04:28.221153 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.211773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33543 (local address 127.31.115.195:57917)
0620 08:04:28.211973 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.212012 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.212525 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.212776 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.212786 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.213056 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:28.213536 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.213541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.214320 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.214321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.215132 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.215134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.215537 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.215538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.215674 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.216240 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:28.216248 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:28.217782 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:28.219573 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:28.219581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:28.219583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:28.219816 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:28.220048 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:28.220048 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:28.220049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:28.220168 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:28.220448 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:28.220456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:28.220610 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.220723 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.220893 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260620 08:04:28.221560 4287 heartbeater.cc:347] Connected to a master server at 127.31.115.254:33543
I20260620 08:04:28.221697 4287 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:28.221839 4287 heartbeater.cc:511] Master 127.31.115.254:33543 requested a full tablet report, sending...
I20260620 08:04:28.222235 3831 ts_manager.cc:194] Registered new tserver with Master: e5ccdc6b79d64e7e99dd260da24ee64a (127.31.115.195:38359)
I20260620 08:04:28.222585 3831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:57917
I20260620 08:04:28.232682 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:04:28.241137 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.234239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.31.115.254:33543)
0620 08:04:28.234289 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.234292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.234433 (+ 141us) server_negotiation.cc:407] Connection header received
0620 08:04:28.234481 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.234482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.234493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.234549 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:28.235239 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.235797 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.236623 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.236828 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.236895 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.238308 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:28.238310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:28.238310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:28.238317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:28.239829 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:28.240173 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:28.240174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:28.240175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:28.240235 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:28.240508 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:28.240509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:28.240510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:28.240686 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:28.240794 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.240943 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.241028 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:28.243075 3831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:28.244875 3831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:28.252154 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.247774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.31.115.194:35503)
0620 08:04:28.247984 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.247989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.248002 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:04:28.248097 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.248100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.248119 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.248197 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:28.248462 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.249699 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.250940 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.251651 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.251736 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.251797 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.252014 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.252055 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260620 08:04:28.252182 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.247831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:42493 (local address 127.0.0.1:60208)
0620 08:04:28.248060 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.248071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.248125 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.248517 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.248518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.248520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:28.248552 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.248553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.249859 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.249860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.250787 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.250790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.251778 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.251779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.251882 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.251900 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.251988 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.252110 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":73,"thread_start_us":71,"threads_started":1}
I20260620 08:04:28.252449 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.248046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.31.115.193:42493)
0620 08:04:28.248294 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.248298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.248307 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:28.248344 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.248346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.248360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.248433 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:28.248660 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.249775 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.250893 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.251656 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.251736 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.251797 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.252323 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.252365 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260620 08:04:28.252180 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.247640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:35503 (local address 127.0.0.1:34308)
0620 08:04:28.247949 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.247958 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.248007 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.248305 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.248309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.248311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:28.248382 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.248384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.249808 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.249809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.250869 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.250871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.251778 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.251779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.251891 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.251900 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.251988 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.252083 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260620 08:04:28.253227 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.247411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:38359 (local address 127.0.0.1:48184)
0620 08:04:28.247542 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.247586 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.247659 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.248398 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.248399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.248402 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:28.248460 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.248462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.249399 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.249400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.250780 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.250783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.252985 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.252986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.253050 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.253054 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.253112 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.253154 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260620 08:04:28.253261 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.247530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.31.115.195:38359)
0620 08:04:28.248150 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.248155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.248166 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:28.248206 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.248208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.248225 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.248304 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:28.248537 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.249287 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.251841 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.252823 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.252887 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.252952 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.253131 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.253183 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":156,"threads_started":1}
I20260620 08:04:28.253279 3955 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3d4f7b8fd74173bb8196816b158332 (DEFAULT_TABLE table=test-table [id=224939422f1343b2aae8b5cfccde3534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:28.253466 4088 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3d4f7b8fd74173bb8196816b158332 (DEFAULT_TABLE table=test-table [id=224939422f1343b2aae8b5cfccde3534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:28.253669 3955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3d4f7b8fd74173bb8196816b158332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:28.253881 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3d4f7b8fd74173bb8196816b158332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:28.254290 4221 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3d4f7b8fd74173bb8196816b158332 (DEFAULT_TABLE table=test-table [id=224939422f1343b2aae8b5cfccde3534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:28.254601 4221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3d4f7b8fd74173bb8196816b158332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:28.256690 4304 tablet_bootstrap.cc:492] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: Bootstrap starting.
I20260620 08:04:28.257313 4304 tablet_bootstrap.cc:654] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:28.257687 4305 tablet_bootstrap.cc:492] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Bootstrap starting.
I20260620 08:04:28.258008 4304 log.cc:824] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:28.258313 4306 tablet_bootstrap.cc:492] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Bootstrap starting.
I20260620 08:04:28.258387 4305 tablet_bootstrap.cc:654] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:28.258955 4306 tablet_bootstrap.cc:654] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:28.259011 4305 log.cc:824] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:28.259649 4306 log.cc:824] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:28.259881 4305 tablet_bootstrap.cc:492] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: No bootstrap required, opened a new log
I20260620 08:04:28.259984 4305 ts_tablet_manager.cc:1397] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:28.260360 4306 tablet_bootstrap.cc:492] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: No bootstrap required, opened a new log
I20260620 08:04:28.260413 4306 ts_tablet_manager.cc:1397] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:28.261487 4305 raft_consensus.cc:348] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.261698 4305 raft_consensus.cc:374] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:28.261745 4305 raft_consensus.cc:729] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93132fbbb2114f8b918bf1238a4d1e64, State: Initialized, Role: FOLLOWER
I20260620 08:04:28.261886 4305 consensus_queue.cc:260] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.261896 4306 raft_consensus.cc:348] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.262055 4304 tablet_bootstrap.cc:492] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: No bootstrap required, opened a new log
I20260620 08:04:28.262049 4306 raft_consensus.cc:374] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:28.262101 4306 raft_consensus.cc:729] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ccdc6b79d64e7e99dd260da24ee64a, State: Initialized, Role: FOLLOWER
I20260620 08:04:28.262131 4304 ts_tablet_manager.cc:1397] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:04:28.262219 4305 ts_tablet_manager.cc:1428] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:04:28.262264 4306 consensus_queue.cc:260] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.262342 4154 heartbeater.cc:503] Master 127.31.115.254:33543 was elected leader, sending a full tablet report...
I20260620 08:04:28.262548 4306 ts_tablet_manager.cc:1428] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:04:28.264122 4304 raft_consensus.cc:348] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.264241 4304 raft_consensus.cc:374] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:28.264261 4304 raft_consensus.cc:729] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba808e99fd134405a36e5e61ed483a09, State: Initialized, Role: FOLLOWER
I20260620 08:04:28.264371 4304 consensus_queue.cc:260] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.264632 4304 ts_tablet_manager.cc:1428] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:04:28.264667 4021 heartbeater.cc:503] Master 127.31.115.254:33543 was elected leader, sending a full tablet report...
I20260620 08:04:28.264787 4287 heartbeater.cc:503] Master 127.31.115.254:33543 was elected leader, sending a full tablet report...
W20260620 08:04:28.300118 4155 tablet.cc:2369] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:28.374032 4022 tablet.cc:2369] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:28.461541 4288 tablet.cc:2369] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:28.489158 4312 raft_consensus.cc:482] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:28.489293 4312 raft_consensus.cc:504] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.489815 4312 leader_election.cc:288] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5ccdc6b79d64e7e99dd260da24ee64a (127.31.115.195:38359), 93132fbbb2114f8b918bf1238a4d1e64 (127.31.115.194:35503)
I20260620 08:04:28.494261 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.490197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:57755 (local address 127.31.115.195:38359)
0620 08:04:28.490263 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.490265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.490416 (+ 151us) server_negotiation.cc:407] Connection header received
0620 08:04:28.490489 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.490490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.490512 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.490566 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:28.490743 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.491539 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.493185 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.493821 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.493900 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.493912 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.494146 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.494185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:04:28.494261 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.490412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:45087 (local address 127.31.115.194:35503)
0620 08:04:28.490463 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.490465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.490475 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:28.490567 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.490568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.490578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.490629 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:28.490803 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.491546 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.493185 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.493868 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.493932 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.493940 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.494146 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.494186 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:04:28.494323 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.490324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:35503 (local address 127.31.115.193:45087)
0620 08:04:28.490405 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.490417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.490475 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.490661 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.490662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.490664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:28.490722 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.490723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.491671 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.491672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.493061 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.493065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.493960 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.493961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.494052 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.494059 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.494124 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.494189 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":15,"thread_start_us":41,"threads_started":1}
I20260620 08:04:28.494599 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.490127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:38359 (local address 127.31.115.193:57755)
0620 08:04:28.490305 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.490317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.490420 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.490578 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.490579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.490581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:28.490672 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.490675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.491671 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.491672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.493061 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:28.493065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.493957 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.493958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:28.494052 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.494057 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.494124 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.494189 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":100,"thread_start_us":82,"threads_started":1}
I20260620 08:04:28.494711 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3d4f7b8fd74173bb8196816b158332" candidate_uuid: "ba808e99fd134405a36e5e61ed483a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93132fbbb2114f8b918bf1238a4d1e64" is_pre_election: true
I20260620 08:04:28.494894 4108 raft_consensus.cc:2454] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba808e99fd134405a36e5e61ed483a09 in term 0.
I20260620 08:04:28.494938 4241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3d4f7b8fd74173bb8196816b158332" candidate_uuid: "ba808e99fd134405a36e5e61ed483a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" is_pre_election: true
I20260620 08:04:28.495128 4241 raft_consensus.cc:2454] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba808e99fd134405a36e5e61ed483a09 in term 0.
I20260620 08:04:28.495131 3911 leader_election.cc:302] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93132fbbb2114f8b918bf1238a4d1e64, ba808e99fd134405a36e5e61ed483a09; no voters:
I20260620 08:04:28.495254 4312 raft_consensus.cc:2781] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:28.495311 4312 raft_consensus.cc:482] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:28.495342 4312 raft_consensus.cc:3037] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:28.496014 4312 raft_consensus.cc:504] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.496147 4312 leader_election.cc:288] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [CANDIDATE]: Term 1 election: Requested vote from peers e5ccdc6b79d64e7e99dd260da24ee64a (127.31.115.195:38359), 93132fbbb2114f8b918bf1238a4d1e64 (127.31.115.194:35503)
I20260620 08:04:28.496359 4108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3d4f7b8fd74173bb8196816b158332" candidate_uuid: "ba808e99fd134405a36e5e61ed483a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93132fbbb2114f8b918bf1238a4d1e64"
I20260620 08:04:28.496353 4241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3d4f7b8fd74173bb8196816b158332" candidate_uuid: "ba808e99fd134405a36e5e61ed483a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a"
I20260620 08:04:28.496428 4108 raft_consensus.cc:3037] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:28.496450 4241 raft_consensus.cc:3037] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:28.497133 4108 raft_consensus.cc:2454] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba808e99fd134405a36e5e61ed483a09 in term 1.
I20260620 08:04:28.497215 4241 raft_consensus.cc:2454] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba808e99fd134405a36e5e61ed483a09 in term 1.
I20260620 08:04:28.497402 3911 leader_election.cc:302] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba808e99fd134405a36e5e61ed483a09, e5ccdc6b79d64e7e99dd260da24ee64a; no voters:
I20260620 08:04:28.497534 4312 raft_consensus.cc:2781] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:28.497735 4312 raft_consensus.cc:686] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [term 1 LEADER]: Becoming Leader. State: Replica: ba808e99fd134405a36e5e61ed483a09, State: Running, Role: LEADER
I20260620 08:04:28.497856 4312 consensus_queue.cc:237] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } }
I20260620 08:04:28.498800 3830 catalog_manager.cc:5495] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba808e99fd134405a36e5e61ed483a09 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "ba808e99fd134405a36e5e61ed483a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba808e99fd134405a36e5e61ed483a09" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 42493 } health_report { overall_health: HEALTHY } } }
I20260620 08:04:28.536180 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.533513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.31.115.193:42493)
0620 08:04:28.533565 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.533568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.533598 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:04:28.533754 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.533755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.533784 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.533847 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:28.533995 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.534500 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.535430 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:28.535639 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:28.535699 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:28.535843 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:28.535945 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:28.536009 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.536087 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.536110 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:04:28.538235 4241 raft_consensus.cc:1261] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 1 FOLLOWER]: Refusing update from remote peer ba808e99fd134405a36e5e61ed483a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:28.538246 4108 raft_consensus.cc:1261] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 1 FOLLOWER]: Refusing update from remote peer ba808e99fd134405a36e5e61ed483a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:28.538663 4312 consensus_queue.cc:1035] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ccdc6b79d64e7e99dd260da24ee64a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:28.538857 4315 consensus_queue.cc:1035] T 8a3d4f7b8fd74173bb8196816b158332 P ba808e99fd134405a36e5e61ed483a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93132fbbb2114f8b918bf1238a4d1e64" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 35503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:28.541368 4318 mvcc.cc:205] Tried to move back new op lower bound from 7298837170329296896 to 7298837170168041472. Current Snapshot: MvccSnapshot[applied={T|T < 7298837170329296896}]
I20260620 08:04:28.541785 4319 mvcc.cc:205] Tried to move back new op lower bound from 7298837170329296896 to 7298837170168041472. Current Snapshot: MvccSnapshot[applied={T|T < 7298837170329296896}]
I20260620 08:04:28.543910 3830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:28.544013 3830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:28.544932 3830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f89e3e1597604fbf8ddc320053e6f4d9: Sending DeleteTablet for 3 replicas of tablet 8a3d4f7b8fd74173bb8196816b158332
I20260620 08:04:28.545343 4088 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3d4f7b8fd74173bb8196816b158332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:28 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:34308
I20260620 08:04:28.545408 4221 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3d4f7b8fd74173bb8196816b158332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:28 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:48184
I20260620 08:04:28.545544 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3893
I20260620 08:04:28.545655 3955 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3d4f7b8fd74173bb8196816b158332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:28 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:60208
I20260620 08:04:28.549824 4329 tablet_replica.cc:330] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: stopping tablet replica
I20260620 08:04:28.549979 4329 raft_consensus.cc:2229] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:28.550040 4328 tablet_replica.cc:330] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: stopping tablet replica
I20260620 08:04:28.550158 4329 raft_consensus.cc:2258] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:28.550169 4328 raft_consensus.cc:2229] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:28.550295 4328 raft_consensus.cc:2258] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:28.550765 4329 ts_tablet_manager.cc:1905] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:28.550779 4328 ts_tablet_manager.cc:1905] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:04:28.551940 3818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:42493 (error 108)
I20260620 08:04:28.552297 4329 ts_tablet_manager.cc:1918] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:28.552390 4329 log.cc:1196] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/8a3d4f7b8fd74173bb8196816b158332
I20260620 08:04:28.552541 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4026
I20260620 08:04:28.552697 4329 ts_tablet_manager.cc:1939] T 8a3d4f7b8fd74173bb8196816b158332 P e5ccdc6b79d64e7e99dd260da24ee64a: Deleting consensus metadata
I20260620 08:04:28.552722 4328 ts_tablet_manager.cc:1918] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:28.552789 4328 log.cc:1196] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/8a3d4f7b8fd74173bb8196816b158332
I20260620 08:04:28.553021 4328 ts_tablet_manager.cc:1939] T 8a3d4f7b8fd74173bb8196816b158332 P 93132fbbb2114f8b918bf1238a4d1e64: Deleting consensus metadata
I20260620 08:04:28.554515 3818 catalog_manager.cc:4841] TS e5ccdc6b79d64e7e99dd260da24ee64a (127.31.115.195:38359): tablet 8a3d4f7b8fd74173bb8196816b158332 (table test-table [id=224939422f1343b2aae8b5cfccde3534]) successfully deleted
I20260620 08:04:28.554579 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.554450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:42493 (local address 127.0.0.1:53630)
0620 08:04:28.554501 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.554547 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 08:04:28.554797 3818 catalog_manager.cc:4584] TS ba808e99fd134405a36e5e61ed483a09 (127.31.115.193:42493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a3d4f7b8fd74173bb8196816b158332: Network error: Client connection negotiation failed: client connection to 127.31.115.193:42493: connect: Connection refused (error 111)
I20260620 08:04:28.557911 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.557800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:35503 (local address 127.0.0.1:41550)
0620 08:04:28.557849 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.557882 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.194:35503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 08:04:28.557897 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4159
I20260620 08:04:28.563489 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 3800
2026-06-20T08:04:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2241 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T08:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:28Z Disabled control of system clock
I20260620 08:04:28.592051 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41703
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:04:28.674433 4334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:28.674643 4334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:28.674665 4334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:28.675836 4334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:28.675889 4334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:28.675908 4334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:28.675932 4334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:28.675945 4334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:28.677312 4334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:28.677518 4334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:28.677857 4334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:28.682727 4340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.682755 4339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.682941 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.683243 4334 server_base.cc:1034] running on GCE node
I20260620 08:04:28.683447 4334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:28.683691 4334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:28.684945 4334 hybrid_clock.cc:648] HybridClock initialized: now 1781942668684922 us; error 31 us; skew 500 ppm
I20260620 08:04:28.686450 4334 webserver.cc:460] Webserver started at http://127.31.115.254:45303/ using document root <none> and password file <none>
I20260620 08:04:28.686761 4334 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:28.686836 4334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:28.686992 4334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:28.688084 4334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "649056da5e7ad925ab35220867776725"
server_key_iv: "28ffd613627dc61e5d29adcd91ea8bf2"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.688452 4334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "649056da5e7ad925ab35220867776725"
server_key_iv: "28ffd613627dc61e5d29adcd91ea8bf2"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.689879 4334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:28.690615 4349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.690790 4334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:28.690899 4334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "649056da5e7ad925ab35220867776725"
server_key_iv: "28ffd613627dc61e5d29adcd91ea8bf2"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.690975 4334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:28.694447 4334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:28.694718 4334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:28.694861 4334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:28.698832 4334 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:41703
I20260620 08:04:28.698860 4401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:41703 every 8 connection(s)
I20260620 08:04:28.699259 4334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:28.699959 4402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:28.702739 4402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d: Bootstrap starting.
I20260620 08:04:28.703326 4402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:28.703655 4402 log.cc:824] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:28.704355 4402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d: No bootstrap required, opened a new log
I20260620 08:04:28.706028 4402 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41703 } }
I20260620 08:04:28.706171 4402 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:28.706195 4402 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0f711a3e6e4f439ba66dfd400f6a2d, State: Initialized, Role: FOLLOWER
I20260620 08:04:28.706374 4402 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41703 } }
I20260620 08:04:28.706465 4402 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:28.706511 4402 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:28.706538 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4334
I20260620 08:04:28.706568 4402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:28.706657 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:28.706828 32207 external_mini_cluster.cc:1428] Setting key 4eba7cf07450f30f811f08224d5d4d0f
I20260620 08:04:28.707230 4402 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41703 } }
I20260620 08:04:28.707340 4402 leader_election.cc:302] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0f711a3e6e4f439ba66dfd400f6a2d; no voters:
I20260620 08:04:28.707528 4402 leader_election.cc:288] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:28.707566 4407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:28.707767 4407 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [term 1 LEADER]: Becoming Leader. State: Replica: bc0f711a3e6e4f439ba66dfd400f6a2d, State: Running, Role: LEADER
I20260620 08:04:28.707837 4402 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:28.707971 4407 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41703 } }
I20260620 08:04:28.708632 4408 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41703 } } }
I20260620 08:04:28.708717 4408 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:28.709172 4409 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0f711a3e6e4f439ba66dfd400f6a2d. Latest consensus state: current_term: 1 leader_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f711a3e6e4f439ba66dfd400f6a2d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41703 } } }
I20260620 08:04:28.709344 4409 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:28.709455 4416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:28.709628 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.707371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.31.115.254:41703)
0620 08:04:28.707637 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.707646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.707682 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:04:28.707810 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.707817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.707860 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.708276 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:28.708879 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:28.708882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:28.708883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:28.708887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:28.708926 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:28.708943 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:28.709262 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.709297 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.709358 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260620 08:04:28.710106 4416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:28.711639 4416 catalog_manager.cc:1347] Generated new cluster ID: b9c47e7ab8f64fe28e0d74fb29e617e5
I20260620 08:04:28.711699 4416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:28.727574 4416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:28.727816 4416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:28.739063 4416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d: Generated new TSK 0
I20260620 08:04:28.739286 4416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:28.745227 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41703
--builtin_ntp_servers=127.31.115.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:04:28.830459 4426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:28.830669 4426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:28.830693 4426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:28.831898 4426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:28.831941 4426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:28.831969 4426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:28.833339 4426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:28.833632 4426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:28.833988 4426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:28.840117 4432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.840123 4435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.840111 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.840909 4426 server_base.cc:1034] running on GCE node
I20260620 08:04:28.841172 4426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:28.841445 4426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:28.842629 4426 hybrid_clock.cc:648] HybridClock initialized: now 1781942668842573 us; error 70 us; skew 500 ppm
I20260620 08:04:28.844072 4426 webserver.cc:460] Webserver started at http://127.31.115.193:34291/ using document root <none> and password file <none>
I20260620 08:04:28.844352 4426 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:28.844401 4426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:28.844606 4426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:28.845996 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "04bf7ace566947618d0896cc13f545ff"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "97b9f2e7082b316d811c73d45d327c13"
server_key_iv: "6484c526e5aa7669125e1a440702c436"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.846406 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "04bf7ace566947618d0896cc13f545ff"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "97b9f2e7082b316d811c73d45d327c13"
server_key_iv: "6484c526e5aa7669125e1a440702c436"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.847994 4426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:28.848893 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.849092 4426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:28.849246 4426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "04bf7ace566947618d0896cc13f545ff"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "97b9f2e7082b316d811c73d45d327c13"
server_key_iv: "6484c526e5aa7669125e1a440702c436"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.849332 4426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:28.852857 4426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:28.853153 4426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:28.853302 4426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:28.853513 4426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:28.853977 4426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:28.854028 4426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.854095 4426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:28.854125 4426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.860045 4426 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:34315
I20260620 08:04:28.860150 4554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:34315 every 8 connection(s)
I20260620 08:04:28.860478 4426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:28.863796 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.861226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:53367 (local address 127.31.115.254:41703)
0620 08:04:28.861288 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.861290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.861535 (+ 245us) server_negotiation.cc:407] Connection header received
0620 08:04:28.862154 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.862155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.862165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.862216 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:28.863451 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:28.863453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:28.863453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:28.863460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:28.863481 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:28.863486 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:28.863551 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.863701 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.863722 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:04:28.864015 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.861211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41703 (local address 127.31.115.193:53367)
0620 08:04:28.861411 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.861445 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.862035 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.862293 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.862297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.862304 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:28.863254 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:28.863254 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:28.863291 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:28.863324 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:28.863563 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:28.863570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:28.863574 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.863697 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.863763 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260620 08:04:28.864781 4555 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41703
I20260620 08:04:28.864869 4555 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:28.865032 4555 heartbeater.cc:511] Master 127.31.115.254:41703 requested a full tablet report, sending...
I20260620 08:04:28.865763 4366 ts_manager.cc:194] Registered new tserver with Master: 04bf7ace566947618d0896cc13f545ff (127.31.115.193:34315)
I20260620 08:04:28.870385 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4426
I20260620 08:04:28.870503 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:28.870685 32207 external_mini_cluster.cc:1428] Setting key bd93d8cd22011b47ab3659fe77185639
I20260620 08:04:28.871722 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41703
--builtin_ntp_servers=127.31.115.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:04:28.955683 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:28.955911 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:28.955937 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:28.957461 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:28.957535 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:28.957566 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:28.959117 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:28.959429 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:28.959823 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:28.965222 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.965276 4563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:28.965243 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:28.965277 4558 server_base.cc:1034] running on GCE node
I20260620 08:04:28.965755 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:28.966041 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:28.967238 4558 hybrid_clock.cc:648] HybridClock initialized: now 1781942668967213 us; error 45 us; skew 500 ppm
I20260620 08:04:28.968674 4558 webserver.cc:460] Webserver started at http://127.31.115.194:45871/ using document root <none> and password file <none>
I20260620 08:04:28.968940 4558 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:28.969013 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:28.969174 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:28.970393 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "aebe843b14504424883443ff4ee9ebe7"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "46756997acbed7c8d246232c321a668f"
server_key_iv: "ce4f97bfaf234fda2ff21d248e43ad4e"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.970793 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "aebe843b14504424883443ff4ee9ebe7"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "46756997acbed7c8d246232c321a668f"
server_key_iv: "ce4f97bfaf234fda2ff21d248e43ad4e"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.972340 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:28.973395 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.973711 4558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:28.973824 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "aebe843b14504424883443ff4ee9ebe7"
format_stamp: "Formatted at 2026-06-20 08:04:28 on dist-test-slave-c4w8"
server_key: "46756997acbed7c8d246232c321a668f"
server_key_iv: "ce4f97bfaf234fda2ff21d248e43ad4e"
server_key_version: "encryptionkey@0"
I20260620 08:04:28.973912 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:28.977396 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:28.977770 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:28.977926 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:28.978147 4558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:28.978473 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:28.978528 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.978600 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:28.978628 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:28.985021 4558 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:41021
I20260620 08:04:28.985086 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:41021 every 8 connection(s)
I20260620 08:04:28.985445 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:28.986785 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4558
I20260620 08:04:28.986871 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:28.987093 32207 external_mini_cluster.cc:1428] Setting key 6c5f43bd8694fde2f86c090618304ca5
I20260620 08:04:28.988101 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41703
--builtin_ntp_servers=127.31.115.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 08:04:28.989291 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.986320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:53899 (local address 127.31.115.254:41703)
0620 08:04:28.986376 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:28.986378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:28.986717 (+ 339us) server_negotiation.cc:407] Connection header received
0620 08:04:28.987384 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:28.987386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:28.987408 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:28.987502 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:28.989005 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:28.989007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:28.989008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:28.989013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:28.989033 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:28.989040 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:28.989123 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:28.989202 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:04:28.989219 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:04:28.989514 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:28.986304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41703 (local address 127.31.115.194:53899)
0620 08:04:28.986495 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:28.986540 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:28.987268 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:28.987816 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:28.987820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:28.987831 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:28.988836 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:28.988837 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:28.988867 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:28.988897 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:28.989120 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:28.989127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:28.989130 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:28.989206 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:04:28.989275 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260620 08:04:28.990031 4687 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41703
I20260620 08:04:28.990154 4687 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:28.990319 4687 heartbeater.cc:511] Master 127.31.115.254:41703 requested a full tablet report, sending...
I20260620 08:04:28.990753 4366 ts_manager.cc:194] Registered new tserver with Master: aebe843b14504424883443ff4ee9ebe7 (127.31.115.194:41021)
W20260620 08:04:29.074064 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:29.074275 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:29.074298 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:29.075438 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:29.075475 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:29.075501 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:29.076862 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:29.077114 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:29.077448 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:29.082747 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.082774 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.082748 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:29.082813 4690 server_base.cc:1034] running on GCE node
I20260620 08:04:29.083206 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:29.083479 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:29.084653 4690 hybrid_clock.cc:648] HybridClock initialized: now 1781942669084640 us; error 29 us; skew 500 ppm
I20260620 08:04:29.086068 4690 webserver.cc:460] Webserver started at http://127.31.115.195:32787/ using document root <none> and password file <none>
I20260620 08:04:29.086320 4690 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:29.086369 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:29.086536 4690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:29.087647 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "9074ca37d9e54ac0a0ccbb676818d8d6"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "01612dbb965b96899733bfd6f3b71a5e"
server_key_iv: "2d0ff1a072fb72994d0665e0368a2623"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.088007 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "9074ca37d9e54ac0a0ccbb676818d8d6"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "01612dbb965b96899733bfd6f3b71a5e"
server_key_iv: "2d0ff1a072fb72994d0665e0368a2623"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.089347 4690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:29.090238 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:29.090440 4690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:04:29.090526 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "9074ca37d9e54ac0a0ccbb676818d8d6"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "01612dbb965b96899733bfd6f3b71a5e"
server_key_iv: "2d0ff1a072fb72994d0665e0368a2623"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.090627 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:29.094012 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:29.094231 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:29.094367 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:29.094558 4690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:29.094798 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:29.094827 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:29.094847 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:29.094859 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:29.100442 4690 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:42977
I20260620 08:04:29.100498 4818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:42977 every 8 connection(s)
I20260620 08:04:29.100839 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:29.103870 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4690
I20260620 08:04:29.103967 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:29.104188 32207 external_mini_cluster.cc:1428] Setting key 2b4b0791bc71bca3bd1995fcd99d3074
I20260620 08:04:29.104195 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.101638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:50819 (local address 127.31.115.254:41703)
0620 08:04:29.101716 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.101718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.101974 (+ 256us) server_negotiation.cc:407] Connection header received
0620 08:04:29.102509 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.102511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.102518 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.102577 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.103875 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.103877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.103877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.103882 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.103899 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.103905 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.104012 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.104115 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.104139 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:04:29.104503 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.101593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41703 (local address 127.31.115.195:50819)
0620 08:04:29.101854 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.101888 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.102390 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.102659 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.102663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.102670 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:29.103666 (+ 996us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:29.103667 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:29.103709 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:29.103747 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:29.104020 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:29.104026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:29.104030 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:29.104124 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 08:04:29.104204 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260620 08:04:29.104955 4819 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41703
I20260620 08:04:29.105060 4819 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:29.105227 4819 heartbeater.cc:511] Master 127.31.115.254:41703 requested a full tablet report, sending...
I20260620 08:04:29.105587 4366 ts_manager.cc:194] Registered new tserver with Master: 9074ca37d9e54ac0a0ccbb676818d8d6 (127.31.115.195:42977)
I20260620 08:04:29.116416 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:04:29.119109 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.117910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.31.115.254:41703)
0620 08:04:29.118007 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.118009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.118098 (+ 89us) server_negotiation.cc:407] Connection header received
0620 08:04:29.118165 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.118166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.118171 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.118241 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.118893 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.118894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.118895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.118899 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.118916 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.118921 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.118984 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.119031 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.119045 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:04:29.120980 4366 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:29.122793 4366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:29.127218 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.125295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41021 (local address 127.0.0.1:50982)
0620 08:04:29.125737 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.125747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.125836 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.126084 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.126092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.126099 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:29.126686 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:29.126687 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:29.126706 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:29.126731 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:29.126994 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:29.127001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:29.127004 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:29.127106 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:04:29.127121 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":108,"threads_started":1}
I20260620 08:04:29.127254 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.125378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.31.115.194:41021)
0620 08:04:29.125651 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.125656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.125811 (+ 155us) server_negotiation.cc:407] Connection header received
0620 08:04:29.125925 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.125935 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.125946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.126040 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.126833 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.126841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.126841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.126846 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.126882 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.126922 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.126991 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.127124 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.127144 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":120,"threads_started":1}
I20260620 08:04:29.127486 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.125575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:42977 (local address 127.0.0.1:35504)
0620 08:04:29.126375 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.126384 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.126458 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.126632 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.126633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.126636 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:29.127113 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:29.127114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:29.127121 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:29.127132 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:29.127366 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:29.127367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:29.127367 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 08:04:29.127416 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:04:29.127430 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":12,"thread_start_us":97,"threads_started":1}
I20260620 08:04:29.127666 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.125781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.31.115.195:42977)
0620 08:04:29.125979 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.126001 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.126465 (+ 464us) server_negotiation.cc:407] Connection header received
0620 08:04:29.126519 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.126521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.126530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.126633 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.127211 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.127217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.127218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.127222 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.127256 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.127282 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.127338 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.127563 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.127579 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":132,"threads_started":1}
I20260620 08:04:29.127955 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.125289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:34315 (local address 127.0.0.1:38542)
0620 08:04:29.125605 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.125674 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.125735 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.126726 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.126727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.126729 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:29.127483 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:29.127484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:29.127492 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:29.127507 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:29.127824 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:29.127824 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:29.127825 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:04:29.127875 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:04:29.127889 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":61,"threads_started":1}
I20260620 08:04:29.127974 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.125407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.31.115.193:34315)
0620 08:04:29.125829 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.125833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.125844 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:29.125871 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.125874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.125885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.125959 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.127615 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.127624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.127625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.127630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.127666 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.127714 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.127758 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.127896 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.127912 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":84,"threads_started":1}
I20260620 08:04:29.128224 4621 tablet_service.cc:1467] Processing CreateTablet for tablet f070e0c24d5947069d48878cc0a977d2 (DEFAULT_TABLE table=test-table [id=a64fca184b5b46c7a85dba6442883203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:29.128652 4621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f070e0c24d5947069d48878cc0a977d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:29.128670 4489 tablet_service.cc:1467] Processing CreateTablet for tablet f070e0c24d5947069d48878cc0a977d2 (DEFAULT_TABLE table=test-table [id=a64fca184b5b46c7a85dba6442883203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:29.129036 4489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f070e0c24d5947069d48878cc0a977d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:29.132447 4833 tablet_bootstrap.cc:492] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Bootstrap starting.
I20260620 08:04:29.132894 4835 tablet_bootstrap.cc:492] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff: Bootstrap starting.
I20260620 08:04:29.133232 4833 tablet_bootstrap.cc:654] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:29.133308 4753 tablet_service.cc:1467] Processing CreateTablet for tablet f070e0c24d5947069d48878cc0a977d2 (DEFAULT_TABLE table=test-table [id=a64fca184b5b46c7a85dba6442883203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:29.133483 4835 tablet_bootstrap.cc:654] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:29.133605 4833 log.cc:824] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:29.133778 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f070e0c24d5947069d48878cc0a977d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:29.133901 4835 log.cc:824] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:29.134698 4835 tablet_bootstrap.cc:492] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff: No bootstrap required, opened a new log
I20260620 08:04:29.134763 4835 ts_tablet_manager.cc:1397] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:04:29.137108 4838 tablet_bootstrap.cc:492] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Bootstrap starting.
I20260620 08:04:29.137487 4835 raft_consensus.cc:348] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.137719 4835 raft_consensus.cc:374] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:29.137768 4835 raft_consensus.cc:729] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bf7ace566947618d0896cc13f545ff, State: Initialized, Role: FOLLOWER
I20260620 08:04:29.137884 4835 consensus_queue.cc:260] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.137934 4838 tablet_bootstrap.cc:654] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:29.137950 4833 tablet_bootstrap.cc:492] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: No bootstrap required, opened a new log
I20260620 08:04:29.138031 4833 ts_tablet_manager.cc:1397] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 08:04:29.138170 4835 ts_tablet_manager.cc:1428] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 08:04:29.138280 4838 log.cc:824] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:29.138310 4555 heartbeater.cc:503] Master 127.31.115.254:41703 was elected leader, sending a full tablet report...
I20260620 08:04:29.139464 4838 tablet_bootstrap.cc:492] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: No bootstrap required, opened a new log
I20260620 08:04:29.139557 4838 ts_tablet_manager.cc:1397] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 08:04:29.140976 4833 raft_consensus.cc:348] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.141166 4833 raft_consensus.cc:374] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:29.141203 4833 raft_consensus.cc:729] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebe843b14504424883443ff4ee9ebe7, State: Initialized, Role: FOLLOWER
I20260620 08:04:29.141326 4833 consensus_queue.cc:260] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.141729 4687 heartbeater.cc:503] Master 127.31.115.254:41703 was elected leader, sending a full tablet report...
I20260620 08:04:29.141717 4833 ts_tablet_manager.cc:1428] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 08:04:29.142447 4838 raft_consensus.cc:348] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.142604 4838 raft_consensus.cc:374] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:29.142655 4838 raft_consensus.cc:729] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9074ca37d9e54ac0a0ccbb676818d8d6, State: Initialized, Role: FOLLOWER
I20260620 08:04:29.142781 4838 consensus_queue.cc:260] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.143055 4838 ts_tablet_manager.cc:1428] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 08:04:29.143095 4819 heartbeater.cc:503] Master 127.31.115.254:41703 was elected leader, sending a full tablet report...
W20260620 08:04:29.236228 4688 tablet.cc:2369] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:29.281773 4842 raft_consensus.cc:482] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:29.281894 4842 raft_consensus.cc:504] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.282356 4842 leader_election.cc:288] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aebe843b14504424883443ff4ee9ebe7 (127.31.115.194:41021), 04bf7ace566947618d0896cc13f545ff (127.31.115.193:34315)
I20260620 08:04:29.283890 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.282598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:34315 (local address 127.31.115.195:57463)
0620 08:04:29.282681 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.282695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.282742 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.282896 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.282897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.282899 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:29.283478 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:29.283479 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:29.283489 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:29.283507 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:29.283720 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:29.283721 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:29.283722 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:04:29.283802 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:04:29.283818 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 08:04:29.283890 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.282723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:57463 (local address 127.31.115.193:34315)
0620 08:04:29.282781 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.282783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.282791 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:29.282819 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.282820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.282825 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.282883 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.283609 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.283610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.283611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.283636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.283656 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.283661 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.283723 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.283807 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.283823 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:29.284197 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.282603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41021 (local address 127.31.115.195:45841)
0620 08:04:29.283108 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.283116 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.283181 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.283326 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.283327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.283329 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:29.283789 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:29.283790 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:29.283796 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:29.283807 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:29.284037 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:29.284037 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:29.284039 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:29.284099 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:04:29.284113 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":311,"thread_start_us":119,"threads_started":1}
I20260620 08:04:29.284219 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.282701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:45841 (local address 127.31.115.194:41021)
0620 08:04:29.282755 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.282758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.283177 (+ 419us) server_negotiation.cc:407] Connection header received
0620 08:04:29.283258 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.283259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.283264 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.283321 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.283912 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.283913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.283914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.283919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.283936 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.283942 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.284005 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.284120 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.284137 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:04:29.284538 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f070e0c24d5947069d48878cc0a977d2" candidate_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aebe843b14504424883443ff4ee9ebe7" is_pre_election: true
I20260620 08:04:29.284536 4509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f070e0c24d5947069d48878cc0a977d2" candidate_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04bf7ace566947618d0896cc13f545ff" is_pre_election: true
I20260620 08:04:29.284716 4641 raft_consensus.cc:2454] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9074ca37d9e54ac0a0ccbb676818d8d6 in term 0.
I20260620 08:04:29.284737 4509 raft_consensus.cc:2454] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9074ca37d9e54ac0a0ccbb676818d8d6 in term 0.
I20260620 08:04:29.284976 4706 leader_election.cc:302] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9074ca37d9e54ac0a0ccbb676818d8d6, aebe843b14504424883443ff4ee9ebe7; no voters:
I20260620 08:04:29.285115 4842 raft_consensus.cc:2781] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:29.285197 4842 raft_consensus.cc:482] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:29.285229 4842 raft_consensus.cc:3037] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:29.286019 4842 raft_consensus.cc:504] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.286190 4842 leader_election.cc:288] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [CANDIDATE]: Term 1 election: Requested vote from peers aebe843b14504424883443ff4ee9ebe7 (127.31.115.194:41021), 04bf7ace566947618d0896cc13f545ff (127.31.115.193:34315)
I20260620 08:04:29.286360 4509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f070e0c24d5947069d48878cc0a977d2" candidate_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04bf7ace566947618d0896cc13f545ff"
I20260620 08:04:29.286424 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f070e0c24d5947069d48878cc0a977d2" candidate_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aebe843b14504424883443ff4ee9ebe7"
I20260620 08:04:29.286453 4509 raft_consensus.cc:3037] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:29.286494 4641 raft_consensus.cc:3037] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:29.287395 4509 raft_consensus.cc:2454] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9074ca37d9e54ac0a0ccbb676818d8d6 in term 1.
I20260620 08:04:29.287395 4641 raft_consensus.cc:2454] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9074ca37d9e54ac0a0ccbb676818d8d6 in term 1.
I20260620 08:04:29.287588 4706 leader_election.cc:302] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9074ca37d9e54ac0a0ccbb676818d8d6, aebe843b14504424883443ff4ee9ebe7; no voters:
I20260620 08:04:29.287693 4842 raft_consensus.cc:2781] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:29.287838 4842 raft_consensus.cc:686] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 1 LEADER]: Becoming Leader. State: Replica: 9074ca37d9e54ac0a0ccbb676818d8d6, State: Running, Role: LEADER
I20260620 08:04:29.287956 4842 consensus_queue.cc:237] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } }
I20260620 08:04:29.288902 4365 catalog_manager.cc:5495] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9074ca37d9e54ac0a0ccbb676818d8d6 (127.31.115.195). New cstate: current_term: 1 leader_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9074ca37d9e54ac0a0ccbb676818d8d6" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42977 } health_report { overall_health: HEALTHY } } }
I20260620 08:04:29.311439 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.310221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.31.115.195:42977)
0620 08:04:29.310324 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.310326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.310336 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:29.310409 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.310410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.310435 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.310507 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.311136 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.311138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.311138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.311144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.311162 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.311174 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.311243 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.311332 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.311350 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 08:04:29.313683 4641 raft_consensus.cc:1261] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 1 FOLLOWER]: Refusing update from remote peer 9074ca37d9e54ac0a0ccbb676818d8d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:29.313944 4509 raft_consensus.cc:1261] T f070e0c24d5947069d48878cc0a977d2 P 04bf7ace566947618d0896cc13f545ff [term 1 FOLLOWER]: Refusing update from remote peer 9074ca37d9e54ac0a0ccbb676818d8d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:29.314213 4844 consensus_queue.cc:1035] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aebe843b14504424883443ff4ee9ebe7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:29.314402 4842 consensus_queue.cc:1035] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04bf7ace566947618d0896cc13f545ff" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:29.317809 4847 mvcc.cc:205] Tried to move back new op lower bound from 7298837173505277952 to 7298837173404327936. Current Snapshot: MvccSnapshot[applied={T|T < 7298837173505277952}]
I20260620 08:04:29.321456 4366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:29.321597 4366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:29.323551 4366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc0f711a3e6e4f439ba66dfd400f6a2d: Sending DeleteTablet for 3 replicas of tablet f070e0c24d5947069d48878cc0a977d2
I20260620 08:04:29.324045 4489 tablet_service.cc:1514] Processing DeleteTablet for tablet f070e0c24d5947069d48878cc0a977d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:29 UTC) from {username='slave'} at 127.0.0.1:38542
I20260620 08:04:29.324294 4621 tablet_service.cc:1514] Processing DeleteTablet for tablet f070e0c24d5947069d48878cc0a977d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:29 UTC) from {username='slave'} at 127.0.0.1:50982
I20260620 08:04:29.324584 4857 tablet_replica.cc:330] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: stopping tablet replica
I20260620 08:04:29.324707 4857 raft_consensus.cc:2229] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:29.324826 4857 raft_consensus.cc:2258] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:29.325350 4857 ts_tablet_manager.cc:1905] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:29.325507 4753 tablet_service.cc:1514] Processing DeleteTablet for tablet f070e0c24d5947069d48878cc0a977d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:29 UTC) from {username='slave'} at 127.0.0.1:35504
I20260620 08:04:29.325762 4858 tablet_replica.cc:330] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: stopping tablet replica
I20260620 08:04:29.326068 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4426
I20260620 08:04:29.327683 4857 ts_tablet_manager.cc:1918] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:29.327767 4857 log.cc:1196] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/f070e0c24d5947069d48878cc0a977d2
I20260620 08:04:29.328042 4857 ts_tablet_manager.cc:1939] T f070e0c24d5947069d48878cc0a977d2 P aebe843b14504424883443ff4ee9ebe7: Deleting consensus metadata
I20260620 08:04:29.328538 4350 catalog_manager.cc:4841] TS aebe843b14504424883443ff4ee9ebe7 (127.31.115.194:41021): tablet f070e0c24d5947069d48878cc0a977d2 (table test-table [id=a64fca184b5b46c7a85dba6442883203]) successfully deleted
W20260620 08:04:29.332845 4351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:34315 (error 108)
I20260620 08:04:29.332886 4858 raft_consensus.cc:2229] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:29.333138 4858 raft_consensus.cc:2258] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:29.333442 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.333288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:34315 (local address 127.0.0.1:38556)
0620 08:04:29.333353 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.333403 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:34315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 08:04:29.333581 4351 catalog_manager.cc:4584] TS 04bf7ace566947618d0896cc13f545ff (127.31.115.193:34315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f070e0c24d5947069d48878cc0a977d2: Network error: Client connection negotiation failed: client connection to 127.31.115.193:34315: connect: Connection refused (error 111)
I20260620 08:04:29.333945 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4558
I20260620 08:04:29.333957 4858 ts_tablet_manager.cc:1905] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:29.336172 4858 ts_tablet_manager.cc:1918] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:29.336294 4858 log.cc:1196] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/f070e0c24d5947069d48878cc0a977d2
I20260620 08:04:29.336637 4858 ts_tablet_manager.cc:1939] T f070e0c24d5947069d48878cc0a977d2 P 9074ca37d9e54ac0a0ccbb676818d8d6: Deleting consensus metadata
I20260620 08:04:29.337163 4350 catalog_manager.cc:4841] TS 9074ca37d9e54ac0a0ccbb676818d8d6 (127.31.115.195:42977): tablet f070e0c24d5947069d48878cc0a977d2 (table test-table [id=a64fca184b5b46c7a85dba6442883203]) successfully deleted
I20260620 08:04:29.340975 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4690
I20260620 08:04:29.346051 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4334
2026-06-20T08:04:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (781 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 08:04:29.366297 32207 mini_oidc.cc:152] Starting JWKS server
I20260620 08:04:29.367496 32207 webserver.cc:460] Webserver started at https://127.0.0.1:41085/ using document root <none> and password file <none>
I20260620 08:04:29.367632 32207 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:04:29.367830 32207 webserver.cc:460] Webserver started at http://127.0.0.1:46243/ using document root <none> and password file <none>
2026-06-20T08:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:29Z Disabled control of system clock
I20260620 08:04:29.376824 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33541
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:33541
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41085/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/testchainca.cert with env {}
W20260620 08:04:29.462738 4865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:29.462955 4865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:29.462980 4865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:29.464170 4865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:29.464208 4865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:29.464221 4865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:29.464234 4865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:29.464247 4865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:04:29.464260 4865 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:04:29.464272 4865 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41085/jwks.json
I20260620 08:04:29.465911 4865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:33541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41085/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:29.466140 4865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:29.466316 4865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:29.471498 4870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.471509 4873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.471509 4871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:29.471705 4865 server_base.cc:1034] running on GCE node
I20260620 08:04:29.471941 4865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:29.472262 4865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:29.473459 4865 hybrid_clock.cc:648] HybridClock initialized: now 1781942669473434 us; error 37 us; skew 500 ppm
I20260620 08:04:29.475003 4865 webserver.cc:460] Webserver started at http://127.31.115.254:45635/ using document root <none> and password file <none>
I20260620 08:04:29.475317 4865 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:29.475369 4865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:29.475529 4865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:29.476639 4865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "0e72fdfdba9d4199b3efa8d817ccfce8"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "7a9b5cbb6b217c5b4f13a26d946c41ad"
server_key_iv: "68f0da1ae6577f7fc945b161c472a338"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.477012 4865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "0e72fdfdba9d4199b3efa8d817ccfce8"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "7a9b5cbb6b217c5b4f13a26d946c41ad"
server_key_iv: "68f0da1ae6577f7fc945b161c472a338"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.478518 4865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:29.479413 4879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:29.479691 4865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:29.479816 4865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "0e72fdfdba9d4199b3efa8d817ccfce8"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "7a9b5cbb6b217c5b4f13a26d946c41ad"
server_key_iv: "68f0da1ae6577f7fc945b161c472a338"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.479888 4865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:29.487313 4865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:29.487586 4865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:29.487690 4865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:29.491827 4865 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:33541
I20260620 08:04:29.491863 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:33541 every 8 connection(s)
I20260620 08:04:29.492247 4865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:29.493460 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:29.495858 4932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8: Bootstrap starting.
I20260620 08:04:29.496435 4932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:29.496694 4932 log.cc:824] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:29.497380 4932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8: No bootstrap required, opened a new log
I20260620 08:04:29.499094 4932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33541 } }
I20260620 08:04:29.499264 4932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:29.499289 4932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e72fdfdba9d4199b3efa8d817ccfce8, State: Initialized, Role: FOLLOWER
I20260620 08:04:29.499398 4932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33541 } }
I20260620 08:04:29.499481 4932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:29.499521 4932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:29.499588 4932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:29.500144 4932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33541 } }
I20260620 08:04:29.500267 4932 leader_election.cc:302] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e72fdfdba9d4199b3efa8d817ccfce8; no voters:
I20260620 08:04:29.500448 4932 leader_election.cc:288] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:29.500528 4935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:29.500696 4935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [term 1 LEADER]: Becoming Leader. State: Replica: 0e72fdfdba9d4199b3efa8d817ccfce8, State: Running, Role: LEADER
I20260620 08:04:29.500764 4932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:29.500804 4935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33541 } }
I20260620 08:04:29.501235 4937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e72fdfdba9d4199b3efa8d817ccfce8. Latest consensus state: current_term: 1 leader_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33541 } } }
I20260620 08:04:29.501348 4937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:29.501322 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e72fdfdba9d4199b3efa8d817ccfce8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33541 } } }
I20260620 08:04:29.501541 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:29.501943 4943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:29.501997 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4865
I20260620 08:04:29.502102 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:29.502348 32207 external_mini_cluster.cc:1428] Setting key 50b17691410b567165398847be466b87
I20260620 08:04:29.502677 4943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:29.504619 4943 catalog_manager.cc:1347] Generated new cluster ID: f63d70a49b42435a98b4d6eeaa993beb
I20260620 08:04:29.504678 4943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:29.508716 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.503022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.31.115.254:33541)
0620 08:04:29.503218 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.503241 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.503272 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:04:29.503488 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.503495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.503568 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.503714 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.503971 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.505232 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.506330 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.507072 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.507215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.507939 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.507944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.507944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.508175 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.508223 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.508239 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.508312 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.508385 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.508472 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260620 08:04:29.509702 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:04:29.517017 4943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:29.517589 4943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:29.523164 4943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e72fdfdba9d4199b3efa8d817ccfce8: Generated new TSK 0
I20260620 08:04:29.523413 4943 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 08:04:29.551429 4962 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 08:04:29.551551 4962 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 08:04:29.551823 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.534683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.31.115.254:33541)
0620 08:04:29.534741 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.534743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.534847 (+ 104us) server_negotiation.cc:407] Connection header received
0620 08:04:29.534934 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.534935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.534949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.535040 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:04:29.535177 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.535976 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.536879 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.537221 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.537295 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.537518 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:04:29.551571 (+ 14053us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 08:04:29.551740 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 08:04:29.551887 4954 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 08:04:29.552258 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:29.534571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33541 (local address 127.0.0.1:37206)
0620 08:04:29.534766 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.534778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.534845 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.535052 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.535053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.535055 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:04:29.535111 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:29.535112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.536109 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.536110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:29.536786 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:29.536790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.537319 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.537321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:29.537374 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.537398 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:04:29.551756 (+ 14358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 08:04:29.551900 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.115.254:33541: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.115.254:33541: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 08:04:29.552815 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4865
2026-06-20T08:04:29Z chronyd exiting
I20260620 08:04:29.590560 32207 test_util.cc:175] -----------------------------------------------
I20260620 08:04:29.590658 32207 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781942645567166-32207-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 08:04:29.591459 32207 mini_oidc.cc:152] Starting JWKS server
I20260620 08:04:29.592201 32207 webserver.cc:460] Webserver started at https://127.0.0.1:44953/ using document root <none> and password file <none>
I20260620 08:04:29.592257 32207 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:04:29.592424 32207 webserver.cc:460] Webserver started at http://127.0.0.1:46027/ using document root <none> and password file <none>
2026-06-20T08:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:29Z Disabled control of system clock
I20260620 08:04:29.600420 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:35753
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:42011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:35753
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44953/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/testchainca.cert with env {}
W20260620 08:04:29.682106 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:29.682335 4969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:29.682359 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:29.683614 4969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:29.683661 4969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:29.683676 4969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:29.683691 4969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:29.683705 4969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:04:29.683718 4969 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:04:29.683729 4969 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44953/jwks.json
I20260620 08:04:29.685345 4969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:35753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:35753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44953/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:29.685654 4969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:29.685904 4969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:29.691459 4974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.691531 4975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.691437 4977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:29.691826 4969 server_base.cc:1034] running on GCE node
I20260620 08:04:29.692042 4969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:29.692301 4969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:29.693441 4969 hybrid_clock.cc:648] HybridClock initialized: now 1781942669693432 us; error 32 us; skew 500 ppm
I20260620 08:04:29.694849 4969 webserver.cc:460] Webserver started at http://127.31.115.254:33793/ using document root <none> and password file <none>
I20260620 08:04:29.695106 4969 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:29.695178 4969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:29.695358 4969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:29.696444 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "655537de95d94bc6a4f5a84847a66bb7"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "f58d77b3078570e3f8cb4d51641df88d"
server_key_iv: "44bdcf2d06203c7d995d25bdc6a2cc8f"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.696822 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "655537de95d94bc6a4f5a84847a66bb7"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "f58d77b3078570e3f8cb4d51641df88d"
server_key_iv: "44bdcf2d06203c7d995d25bdc6a2cc8f"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.698226 4969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:04:29.699040 4983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:29.699244 4969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:29.699348 4969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "655537de95d94bc6a4f5a84847a66bb7"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "f58d77b3078570e3f8cb4d51641df88d"
server_key_iv: "44bdcf2d06203c7d995d25bdc6a2cc8f"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.699429 4969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:29.716807 4969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:29.717162 4969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:29.717332 4969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:29.721639 4969 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:35753
I20260620 08:04:29.721732 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:35753 every 8 connection(s)
I20260620 08:04:29.722144 4969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:29.722939 5036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:29.724977 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 4969
I20260620 08:04:29.725086 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:29.725277 32207 external_mini_cluster.cc:1428] Setting key dfa75d992daf5ac9d2e1677b4e37d2a7
I20260620 08:04:29.726301 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7: Bootstrap starting.
I20260620 08:04:29.727023 5036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:29.727370 5036 log.cc:824] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:29.728184 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7: No bootstrap required, opened a new log
I20260620 08:04:29.730170 5036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655537de95d94bc6a4f5a84847a66bb7" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35753 } }
I20260620 08:04:29.730343 5036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:29.730391 5036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655537de95d94bc6a4f5a84847a66bb7, State: Initialized, Role: FOLLOWER
I20260620 08:04:29.730535 5036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655537de95d94bc6a4f5a84847a66bb7" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35753 } }
I20260620 08:04:29.730602 5036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:29.730667 5036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:29.730721 5036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:29.731185 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.725896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.31.115.254:35753)
0620 08:04:29.726120 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.726128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.726156 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:04:29.726638 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.726646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.726712 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.726835 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.727119 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.728296 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.729115 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.729535 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.729703 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.730394 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.730399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.730399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.730617 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.730661 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.730677 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.730743 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.730872 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.730963 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260620 08:04:29.731420 5036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655537de95d94bc6a4f5a84847a66bb7" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35753 } }
I20260620 08:04:29.731554 5036 leader_election.cc:302] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 655537de95d94bc6a4f5a84847a66bb7; no voters:
I20260620 08:04:29.731722 5036 leader_election.cc:288] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:29.731742 5041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:29.731854 5041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 655537de95d94bc6a4f5a84847a66bb7, State: Running, Role: LEADER
I20260620 08:04:29.731927 5041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655537de95d94bc6a4f5a84847a66bb7" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35753 } }
I20260620 08:04:29.731992 5036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:29.732096 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:04:29.732247 5042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "655537de95d94bc6a4f5a84847a66bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655537de95d94bc6a4f5a84847a66bb7" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35753 } } }
I20260620 08:04:29.732319 5042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:29.732475 5043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 655537de95d94bc6a4f5a84847a66bb7. Latest consensus state: current_term: 1 leader_uuid: "655537de95d94bc6a4f5a84847a66bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655537de95d94bc6a4f5a84847a66bb7" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35753 } } }
I20260620 08:04:29.732518 5043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:29.732642 5046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:29.733163 5046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:29.734958 5046 catalog_manager.cc:1347] Generated new cluster ID: 108fdb4173264823bda61490ae9a3ca2
I20260620 08:04:29.735009 5046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:29.751129 5046 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:29.751727 5046 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:29.759224 5046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 655537de95d94bc6a4f5a84847a66bb7: Generated new TSK 0
I20260620 08:04:29.759485 5046 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 08:04:29.791072 5066 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 08:04:29.791180 5066 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 08:04:29.791544 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.769494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.31.115.254:35753)
0620 08:04:29.769547 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.769550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.769697 (+ 147us) server_negotiation.cc:407] Connection header received
0620 08:04:29.769776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.769777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.769790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.769853 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:04:29.770026 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.770557 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.771700 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.771907 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.771975 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.772165 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:04:29.791300 (+ 19135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 08:04:29.791468 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 08:04:29.791612 5039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 08:04:29.791733 5065 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:29.769385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:35753 (local address 127.0.0.1:43146)
0620 08:04:29.769580 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:29.769592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:29.769686 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:29.769874 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:29.769876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:29.769878 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:04:29.769952 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:29.769955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.770633 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.770634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:29.771600 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:29.771603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.771989 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.771991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:29.772054 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.772061 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:04:29.791431 (+ 19370us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 08:04:29.791573 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.115.254:35753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":119,"threads_started":1}
I20260620 08:04:29.791913 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 4969
2026-06-20T08:04:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 08:04:29.830252 32207 mini_oidc.cc:152] Starting JWKS server
I20260620 08:04:29.830974 32207 webserver.cc:460] Webserver started at https://127.0.0.1:35871/ using document root <none> and password file <none>
I20260620 08:04:29.831244 32207 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:04:29.831456 32207 webserver.cc:460] Webserver started at http://127.0.0.1:45631/ using document root <none> and password file <none>
2026-06-20T08:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:29Z Disabled control of system clock
I20260620 08:04:29.839720 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40887
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40887
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35871/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 08:04:29.924597 5072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:29.924841 5072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:29.924867 5072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:29.926119 5072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:29.926172 5072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:29.926185 5072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:29.926198 5072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:29.926211 5072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:04:29.926224 5072 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:04:29.926235 5072 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35871/jwks.json
I20260620 08:04:29.927623 5072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35871/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:29.927853 5072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:29.928102 5072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:29.933277 5080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.933293 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:29.933351 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:29.933548 5072 server_base.cc:1034] running on GCE node
I20260620 08:04:29.933840 5072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:29.934149 5072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:29.935357 5072 hybrid_clock.cc:648] HybridClock initialized: now 1781942669935342 us; error 30 us; skew 500 ppm
I20260620 08:04:29.936913 5072 webserver.cc:460] Webserver started at http://127.31.115.254:38377/ using document root <none> and password file <none>
I20260620 08:04:29.937193 5072 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:29.937275 5072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:29.937446 5072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:29.938752 5072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "b79208b88b3b4e639b554d23fbb6dfc3"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "0ca44fee66c4148e01d0d86daec082df"
server_key_iv: "188356c7ebdccac9b37e745ca8d5b693"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.939167 5072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "b79208b88b3b4e639b554d23fbb6dfc3"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "0ca44fee66c4148e01d0d86daec082df"
server_key_iv: "188356c7ebdccac9b37e745ca8d5b693"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.940809 5072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:29.941745 5086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:29.941947 5072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:29.942052 5072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "b79208b88b3b4e639b554d23fbb6dfc3"
format_stamp: "Formatted at 2026-06-20 08:04:29 on dist-test-slave-c4w8"
server_key: "0ca44fee66c4148e01d0d86daec082df"
server_key_iv: "188356c7ebdccac9b37e745ca8d5b693"
server_key_version: "encryptionkey@0"
I20260620 08:04:29.942147 5072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:29.964340 5072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:29.964740 5072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:29.964913 5072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:29.969182 5072 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:40887
I20260620 08:04:29.969252 5139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:40887 every 8 connection(s)
I20260620 08:04:29.969637 5072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:29.970376 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:29.973124 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3: Bootstrap starting.
I20260620 08:04:29.973802 5140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:29.974056 5140 log.cc:824] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:29.974687 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5072
I20260620 08:04:29.974792 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:29.974959 32207 external_mini_cluster.cc:1428] Setting key 268e65c44cee3ea42bfaf24784eaa8f5
I20260620 08:04:29.974962 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3: No bootstrap required, opened a new log
I20260620 08:04:29.976924 5140 raft_consensus.cc:348] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40887 } }
I20260620 08:04:29.977068 5140 raft_consensus.cc:374] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:29.977092 5140 raft_consensus.cc:729] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79208b88b3b4e639b554d23fbb6dfc3, State: Initialized, Role: FOLLOWER
I20260620 08:04:29.977264 5140 consensus_queue.cc:260] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40887 } }
I20260620 08:04:29.977363 5140 raft_consensus.cc:388] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:29.977398 5140 raft_consensus.cc:482] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:29.977453 5140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:29.978094 5140 raft_consensus.cc:504] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40887 } }
I20260620 08:04:29.978226 5140 leader_election.cc:302] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b79208b88b3b4e639b554d23fbb6dfc3; no voters:
I20260620 08:04:29.978441 5140 leader_election.cc:288] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:29.978466 5145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:29.978575 5145 raft_consensus.cc:686] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [term 1 LEADER]: Becoming Leader. State: Replica: b79208b88b3b4e639b554d23fbb6dfc3, State: Running, Role: LEADER
I20260620 08:04:29.978672 5145 consensus_queue.cc:237] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40887 } }
I20260620 08:04:29.978753 5140 sys_catalog.cc:564] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:29.979008 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b79208b88b3b4e639b554d23fbb6dfc3. Latest consensus state: current_term: 1 leader_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40887 } } }
I20260620 08:04:29.979238 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:29.979156 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79208b88b3b4e639b554d23fbb6dfc3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40887 } } }
I20260620 08:04:29.979497 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:29.979660 5153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:29.980203 5153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:29.982255 5153 catalog_manager.cc:1347] Generated new cluster ID: e4a53c240a1b45d38ad16d8bf40142ff
I20260620 08:04:29.982317 5153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:29.982940 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:29.975510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.31.115.254:40887)
0620 08:04:29.975717 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:29.975725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:29.975750 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:04:29.976914 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:29.976923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:29.976992 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:29.977185 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:29.977521 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.979178 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.980305 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:29.980925 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:29.981044 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:29.982037 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:29.982041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:29.982041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:29.982283 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:29.982350 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:29.982368 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:29.982526 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:29.982671 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 08:04:29.982762 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260620 08:04:29.984294 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:04:29.987126 5153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:29.987799 5153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:29.993968 5153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b79208b88b3b4e639b554d23fbb6dfc3: Generated new TSK 0
I20260620 08:04:29.994189 5153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:30.008433 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:30.001947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.31.115.254:40887)
0620 08:04:30.002064 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:30.002066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:30.002323 (+ 257us) server_negotiation.cc:407] Connection header received
0620 08:04:30.002575 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:30.002577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:30.002705 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:30.002826 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:04:30.003113 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.003745 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.005360 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.005774 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.005851 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:30.006348 (+ 497us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:04:30.008109 (+ 1761us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35871/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 08:04:30.008349 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35871/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 08:04:30.008522 5144 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35871/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 08:04:30.011780 5169 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:30.001837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40887 (local address 127.0.0.1:57018)
0620 08:04:30.002219 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:30.002233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:30.002317 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:30.002826 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:30.002827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:30.002830 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:04:30.002897 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:30.002900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.003873 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.003875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:30.005120 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:30.005124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.005878 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.005880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:30.005969 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:30.005977 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:04:30.008286 (+ 2309us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35871/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 08:04:30.011681 (+ 3395us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.115.254:40887: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35871/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3169,"spinlock_wait_cycles":25344,"thread_start_us":124,"threads_started":1}
I20260620 08:04:30.012269 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5072
E20260620 08:04:30.022236 5170 webserver.cc:533] Webserver: sslize failed:
E20260620 08:04:30.022331 5170 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T08:04:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T08:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:30Z Disabled control of system clock
I20260620 08:04:30.059719 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33493
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:33493
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/testchainca.cert with env {}
W20260620 08:04:30.142725 5174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:30.142940 5174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:30.142962 5174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:30.144269 5174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:30.144311 5174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:30.144325 5174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:30.144337 5174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:30.144349 5174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:04:30.144361 5174 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:04:30.144415 5174 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 08:04:30.145979 5174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:33493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:30.146242 5174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:30.146469 5174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:30.151664 5179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:30.151652 5182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:30.151655 5180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:30.151686 5174 server_base.cc:1034] running on GCE node
I20260620 08:04:30.152154 5174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:30.152431 5174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:30.153751 5174 hybrid_clock.cc:648] HybridClock initialized: now 1781942670153703 us; error 53 us; skew 500 ppm
I20260620 08:04:30.155102 5174 webserver.cc:460] Webserver started at http://127.31.115.254:44443/ using document root <none> and password file <none>
I20260620 08:04:30.155364 5174 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:30.155416 5174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:30.155550 5174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:30.156648 5174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "dab818d62cfb4a3686dc9161ed02361a"
format_stamp: "Formatted at 2026-06-20 08:04:30 on dist-test-slave-c4w8"
server_key: "85e8aee5bbb1a9812b7e7d44d993f0df"
server_key_iv: "2a5b2d79fb5ceb0467fc7f03a2fd23a9"
server_key_version: "encryptionkey@0"
I20260620 08:04:30.157007 5174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "dab818d62cfb4a3686dc9161ed02361a"
format_stamp: "Formatted at 2026-06-20 08:04:30 on dist-test-slave-c4w8"
server_key: "85e8aee5bbb1a9812b7e7d44d993f0df"
server_key_iv: "2a5b2d79fb5ceb0467fc7f03a2fd23a9"
server_key_version: "encryptionkey@0"
I20260620 08:04:30.158581 5174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:30.159471 5188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:30.159731 5174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:30.159829 5174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "dab818d62cfb4a3686dc9161ed02361a"
format_stamp: "Formatted at 2026-06-20 08:04:30 on dist-test-slave-c4w8"
server_key: "85e8aee5bbb1a9812b7e7d44d993f0df"
server_key_iv: "2a5b2d79fb5ceb0467fc7f03a2fd23a9"
server_key_version: "encryptionkey@0"
I20260620 08:04:30.159888 5174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:30.174363 5174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:30.174651 5174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:30.174758 5174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:30.178736 5174 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:33493
I20260620 08:04:30.178833 5241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:33493 every 8 connection(s)
I20260620 08:04:30.179149 5174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:30.180348 5242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:30.182966 5242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a: Bootstrap starting.
I20260620 08:04:30.183606 5242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:30.183925 5242 log.cc:824] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:30.184383 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5174
I20260620 08:04:30.184504 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:30.184748 32207 external_mini_cluster.cc:1428] Setting key afc284cf919b83ab0154576ef3b9daf5
I20260620 08:04:30.184765 5242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a: No bootstrap required, opened a new log
I20260620 08:04:30.187716 5242 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab818d62cfb4a3686dc9161ed02361a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33493 } }
I20260620 08:04:30.187889 5242 raft_consensus.cc:374] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:30.187919 5242 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab818d62cfb4a3686dc9161ed02361a, State: Initialized, Role: FOLLOWER
I20260620 08:04:30.188035 5242 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab818d62cfb4a3686dc9161ed02361a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33493 } }
I20260620 08:04:30.188122 5242 raft_consensus.cc:388] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:30.188162 5242 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:30.188220 5242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:30.188787 5242 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab818d62cfb4a3686dc9161ed02361a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33493 } }
I20260620 08:04:30.188917 5242 leader_election.cc:302] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab818d62cfb4a3686dc9161ed02361a; no voters:
I20260620 08:04:30.189085 5242 leader_election.cc:288] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:30.189162 5247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:30.189308 5247 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [term 1 LEADER]: Becoming Leader. State: Replica: dab818d62cfb4a3686dc9161ed02361a, State: Running, Role: LEADER
I20260620 08:04:30.189337 5242 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:30.189395 5247 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab818d62cfb4a3686dc9161ed02361a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33493 } }
I20260620 08:04:30.189756 5248 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dab818d62cfb4a3686dc9161ed02361a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab818d62cfb4a3686dc9161ed02361a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33493 } } }
I20260620 08:04:30.189849 5248 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:30.189743 5247 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab818d62cfb4a3686dc9161ed02361a. Latest consensus state: current_term: 1 leader_uuid: "dab818d62cfb4a3686dc9161ed02361a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab818d62cfb4a3686dc9161ed02361a" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33493 } } }
I20260620 08:04:30.190047 5247 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:30.190152 5253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:30.190719 5253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:30.191633 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:30.185303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.31.115.254:33493)
0620 08:04:30.185547 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:30.185557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:30.185595 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:04:30.186256 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:30.186266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:30.186347 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:30.186503 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:30.186678 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.188083 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.188887 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.189525 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.190546 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:30.190738 (+ 192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:30.190742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:30.190743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:30.190973 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:30.191017 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:30.191048 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:30.191150 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:30.191240 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 08:04:30.191421 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260620 08:04:30.192632 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:04:30.198190 5253 catalog_manager.cc:1347] Generated new cluster ID: 21f6a4e974364046b01b3eba1a47d89d
I20260620 08:04:30.198246 5253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:30.211993 5253 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:30.212643 5253 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:30.217454 5253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dab818d62cfb4a3686dc9161ed02361a: Generated new TSK 0
I20260620 08:04:30.217789 5253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:30.298887 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:30.231896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.31.115.254:33493)
0620 08:04:30.231968 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:30.231970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:30.232095 (+ 125us) server_negotiation.cc:407] Connection header received
0620 08:04:30.232178 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:30.232179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:30.232191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:30.232257 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:04:30.232401 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.232913 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.234094 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.234337 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.234402 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:30.234633 (+ 231us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:04:30.298653 (+ 64020us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 08:04:30.298820 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48924: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 08:04:30.299001 5246 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48924: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 08:04:30.299080 5271 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:30.231835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33493 (local address 127.0.0.1:48924)
0620 08:04:30.232017 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:30.232029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:30.232101 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:30.232271 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:30.232272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:30.232274 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:04:30.232339 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:30.232341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.232999 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.233000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:30.233988 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:30.233993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:30.234426 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:30.234427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:30.234494 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:30.234502 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:04:30.298810 (+ 64308us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 08:04:30.298968 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.115.254:33493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":127,"threads_started":1}
I20260620 08:04:30.299556 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5174
2026-06-20T08:04:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (266 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:30 dist-test-slave-c4w8 krb5kdc[5283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:30 dist-test-slave-c4w8 krb5kdc[5283](info): set up 2 sockets
Jun 20 08:04:30 dist-test-slave-c4w8 krb5kdc[5283](info): commencing operation
krb5kdc: starting...
W20260620 08:04:31.546442 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:31 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:31.673252 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41577
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:31.758514 5296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:31.758809 5296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:31.758853 5296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:31.760154 5296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:31.760238 5296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:31.760262 5296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:31.760282 5296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:31.760313 5296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:31.761966 5296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:31.762300 5296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:31.762557 5296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:31.768303 5302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:31.768265 5301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:31.768430 5296 server_base.cc:1034] running on GCE node
W20260620 08:04:31.768265 5304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:31.768882 5296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:31.769181 5296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:31.770395 5296 hybrid_clock.cc:648] HybridClock initialized: now 1781942671770362 us; error 63 us; skew 500 ppm
Jun 20 08:04:31 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942671, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:31.773263 5296 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:31.773909 5296 webserver.cc:460] Webserver started at http://127.31.115.254:36339/ using document root <none> and password file <none>
I20260620 08:04:31.774179 5296 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:31.774255 5296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:31.774420 5296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:31.775650 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "c1086bf507064727b2dab004332da1d2"
format_stamp: "Formatted at 2026-06-20 08:04:31 on dist-test-slave-c4w8"
server_key: "57d62b08ff3e270eeecce4aaa3158c7b"
server_key_iv: "1b477f63eb17676ec5a77f983f13700f"
server_key_version: "encryptionkey@0"
I20260620 08:04:31.776078 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "c1086bf507064727b2dab004332da1d2"
format_stamp: "Formatted at 2026-06-20 08:04:31 on dist-test-slave-c4w8"
server_key: "57d62b08ff3e270eeecce4aaa3158c7b"
server_key_iv: "1b477f63eb17676ec5a77f983f13700f"
server_key_version: "encryptionkey@0"
I20260620 08:04:31.777805 5296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:31.778882 5311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:31.779194 5296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:31.779331 5296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "c1086bf507064727b2dab004332da1d2"
format_stamp: "Formatted at 2026-06-20 08:04:31 on dist-test-slave-c4w8"
server_key: "57d62b08ff3e270eeecce4aaa3158c7b"
server_key_iv: "1b477f63eb17676ec5a77f983f13700f"
server_key_version: "encryptionkey@0"
I20260620 08:04:31.779417 5296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:31.819028 5296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:31.821498 5296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:31.821722 5296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:31.826005 5296 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:41577
I20260620 08:04:31.826150 5363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:41577 every 8 connection(s)
I20260620 08:04:31.826488 5296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:31.827523 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:31.829770 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5296
I20260620 08:04:31.829893 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:31.830092 32207 external_mini_cluster.cc:1428] Setting key 7dfc0122d5140d24c4e6ce80893fa651
I20260620 08:04:31.830921 5364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2: Bootstrap starting.
I20260620 08:04:31.831759 5364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:31.832144 5364 log.cc:824] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:31.833130 5364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2: No bootstrap required, opened a new log
I20260620 08:04:31.835884 5364 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1086bf507064727b2dab004332da1d2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41577 } }
I20260620 08:04:31.836087 5364 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:31.836133 5364 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1086bf507064727b2dab004332da1d2, State: Initialized, Role: FOLLOWER
I20260620 08:04:31.836280 5364 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1086bf507064727b2dab004332da1d2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41577 } }
I20260620 08:04:31.836381 5364 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:31.836431 5364 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:31.836484 5364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:31.837118 5364 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1086bf507064727b2dab004332da1d2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41577 } }
I20260620 08:04:31.837255 5364 leader_election.cc:302] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1086bf507064727b2dab004332da1d2; no voters:
I20260620 08:04:31.837482 5364 leader_election.cc:288] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:31.837517 5369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:31.837663 5369 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [term 1 LEADER]: Becoming Leader. State: Replica: c1086bf507064727b2dab004332da1d2, State: Running, Role: LEADER
I20260620 08:04:31.837833 5369 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1086bf507064727b2dab004332da1d2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41577 } }
I20260620 08:04:31.837863 5364 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:31.838292 5371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1086bf507064727b2dab004332da1d2. Latest consensus state: current_term: 1 leader_uuid: "c1086bf507064727b2dab004332da1d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1086bf507064727b2dab004332da1d2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41577 } } }
I20260620 08:04:31.838373 5371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [sys.catalog]: This master's current role is: LEADER
Jun 20 08:04:31 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:31.838608 5370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1086bf507064727b2dab004332da1d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1086bf507064727b2dab004332da1d2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41577 } } }
I20260620 08:04:31.838905 5370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:31.839046 5374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:31.839627 5374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:31.841584 5374 catalog_manager.cc:1347] Generated new cluster ID: 07c72e1cdd9e48d28c5cdfbfbdc172a1
I20260620 08:04:31.841678 5374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:31.843737 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:31.830731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.31.115.254:41577)
0620 08:04:31.830958 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:31.830970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:31.831009 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:04:31.831705 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:31.831717 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:31.831797 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:31.831984 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:31.832964 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:31.834578 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:31.835392 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:31.836105 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:31.836239 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:31.839533 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:31.839538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:31.839538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:31.839553 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:31.841933 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:31.842359 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:31.842363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:31.842363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:31.842480 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:31.842723 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:31.842724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:31.842724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:31.843044 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:31.843201 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:31.843349 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 08:04:31.843495 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260620 08:04:31.872283 5374 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:31.872984 5374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:31.879467 5374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1086bf507064727b2dab004332da1d2: Generated new TSK 0
I20260620 08:04:31.879789 5374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:31.913591 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41577
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:32.003613 5390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:32.003880 5390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:32.003926 5390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:32.005170 5390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:32.005247 5390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:32.005293 5390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:32.007047 5390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:32.007519 5390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:32.007826 5390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:32.013808 5396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:32.013823 5395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:32.013808 5398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:32.014099 5390 server_base.cc:1034] running on GCE node
I20260620 08:04:32.014319 5390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:32.014567 5390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:32.015739 5390 hybrid_clock.cc:648] HybridClock initialized: now 1781942672015722 us; error 34 us; skew 500 ppm
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.018699 5390 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:32.019248 5390 webserver.cc:460] Webserver started at http://127.31.115.193:34057/ using document root <none> and password file <none>
I20260620 08:04:32.019508 5390 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:32.019558 5390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:32.019701 5390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:32.020838 5390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "29e0a4a6f9e745109c67ff18d8cb1ee1"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "3a2e17a1586b3b50381e166d300ca2e8"
server_key_iv: "9c5c332d88227f9f0efc5ec1e3f5daca"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.021220 5390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "29e0a4a6f9e745109c67ff18d8cb1ee1"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "3a2e17a1586b3b50381e166d300ca2e8"
server_key_iv: "9c5c332d88227f9f0efc5ec1e3f5daca"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.022940 5390 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 08:04:32.023977 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.024297 5390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:32.024389 5390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "29e0a4a6f9e745109c67ff18d8cb1ee1"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "3a2e17a1586b3b50381e166d300ca2e8"
server_key_iv: "9c5c332d88227f9f0efc5ec1e3f5daca"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.024492 5390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:32.052977 5390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:32.055745 5390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:32.055953 5390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:32.056322 5390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:32.056794 5390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:32.056845 5390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.056872 5390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:32.056887 5390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.064131 5390 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:39303
I20260620 08:04:32.064218 5518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:39303 every 8 connection(s)
I20260620 08:04:32.064589 5390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:32.070056 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5390
I20260620 08:04:32.070185 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:32.070401 32207 external_mini_cluster.cc:1428] Setting key 10043d8b7241117a12343c471a2688c2
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:32.075570 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.065595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:33913 (local address 127.31.115.254:41577)
0620 08:04:32.065686 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.065691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.065888 (+ 197us) server_negotiation.cc:407] Connection header received
0620 08:04:32.066852 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.066853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.066866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.066930 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.067593 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.068148 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.069233 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.069454 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.069530 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.072057 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:32.072059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:32.072059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:32.072068 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:32.073879 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.074453 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.074454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.074454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.074509 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.074821 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.074822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.074822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.074933 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:32.074984 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:32.075388 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 08:04:32.075500 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 08:04:32.075855 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.065499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41577 (local address 127.31.115.193:33913)
0620 08:04:32.065698 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:32.065742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:32.066715 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:32.066988 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:32.066991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:32.067262 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:32.067462 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:32.067467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.068277 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.068279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:32.069130 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:32.069133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.069558 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.069559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:32.069774 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.070407 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:32.070416 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:32.071962 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:32.073972 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:32.073982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:32.073984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:32.074357 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:32.074580 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:32.074581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:32.074582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:32.074746 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:32.075007 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:32.075015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:32.075209 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 08:04:32.075348 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 08:04:32.075557 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260620 08:04:32.076489 5519 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41577
I20260620 08:04:32.076620 5519 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:32.076859 5519 heartbeater.cc:511] Master 127.31.115.254:41577 requested a full tablet report, sending...
I20260620 08:04:32.077468 5328 ts_manager.cc:194] Registered new tserver with Master: 29e0a4a6f9e745109c67ff18d8cb1ee1 (127.31.115.193:39303)
I20260620 08:04:32.078174 5328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:33913
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:32.101342 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41577
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:32.190092 5524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:32.190335 5524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:32.190375 5524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:32.191571 5524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:32.191658 5524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:32.191708 5524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:32.193233 5524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:32.193511 5524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:32.193774 5524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:04:32.199165 5524 server_base.cc:1034] running on GCE node
W20260620 08:04:32.199119 5530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:32.199131 5532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:32.199131 5529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:32.199484 5524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:32.199801 5524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:32.200986 5524 hybrid_clock.cc:648] HybridClock initialized: now 1781942672200965 us; error 37 us; skew 500 ppm
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.203637 5524 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:32.204183 5524 webserver.cc:460] Webserver started at http://127.31.115.194:39471/ using document root <none> and password file <none>
I20260620 08:04:32.204420 5524 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:32.204489 5524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:32.204643 5524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:32.205793 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "a03985fa1fb948e48fdb9fb64f9f0274"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "14aa6ea6cb7755be62e29559ef889346"
server_key_iv: "f4cd379ef67dc1715ae89b8b1d6380e8"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.206173 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "a03985fa1fb948e48fdb9fb64f9f0274"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "14aa6ea6cb7755be62e29559ef889346"
server_key_iv: "f4cd379ef67dc1715ae89b8b1d6380e8"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.207623 5524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:32.208776 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.209066 5524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:32.209205 5524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "a03985fa1fb948e48fdb9fb64f9f0274"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "14aa6ea6cb7755be62e29559ef889346"
server_key_iv: "f4cd379ef67dc1715ae89b8b1d6380e8"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.209296 5524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:32.236110 5524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:32.238922 5524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:32.239089 5524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:32.239353 5524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:32.239720 5524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:32.239773 5524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.239823 5524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:32.239843 5524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.245724 5524 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:44085
I20260620 08:04:32.245813 5652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:44085 every 8 connection(s)
I20260620 08:04:32.246157 5524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:32.247123 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5524
I20260620 08:04:32.247259 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:32.247466 32207 external_mini_cluster.cc:1428] Setting key 3e80448ce15d7f9448c8bf73c5a2b96c
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:04:32.256727 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.247011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:33439 (local address 127.31.115.254:41577)
0620 08:04:32.247090 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.247092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.247246 (+ 154us) server_negotiation.cc:407] Connection header received
0620 08:04:32.248461 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.248462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.248474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.248539 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.249156 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.249912 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.250950 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.251176 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.251257 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.253810 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:32.253812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:32.253812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:32.253820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:32.255264 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.255736 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.255738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.255739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.255806 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.256053 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.256054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.256055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.256181 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:32.256241 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:32.256488 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 08:04:32.256556 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:32.256834 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.246927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41577 (local address 127.31.115.194:33439)
0620 08:04:32.247101 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:32.247152 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:32.248362 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:32.248610 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:32.248613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:32.248869 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:32.249065 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:32.249068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.250023 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.250024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:32.250852 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:32.250855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.251265 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.251267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:32.251405 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.251953 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:32.251962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:32.253698 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:32.255372 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:32.255379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:32.255382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:32.255619 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:32.255887 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:32.255888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:32.255888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:32.255991 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:32.256245 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:32.256249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:32.256374 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 08:04:32.256460 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:04:32.256586 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260620 08:04:32.257452 5653 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41577
I20260620 08:04:32.257625 5653 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:32.257884 5653 heartbeater.cc:511] Master 127.31.115.254:41577 requested a full tablet report, sending...
I20260620 08:04:32.258414 5328 ts_manager.cc:194] Registered new tserver with Master: a03985fa1fb948e48fdb9fb64f9f0274 (127.31.115.194:44085)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
I20260620 08:04:32.258877 5328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:33439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:32.276160 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41577
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:32.361141 5658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:32.361353 5658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:32.361379 5658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:32.362596 5658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:32.362643 5658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:32.362670 5658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:32.364207 5658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:32.364430 5658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:32.364684 5658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:32.370118 5663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:32.370119 5666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:32.370235 5658 server_base.cc:1034] running on GCE node
W20260620 08:04:32.370119 5664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:32.370658 5658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:32.370939 5658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:32.372110 5658 hybrid_clock.cc:648] HybridClock initialized: now 1781942672372075 us; error 52 us; skew 500 ppm
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.375370 5658 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:32.375921 5658 webserver.cc:460] Webserver started at http://127.31.115.195:44513/ using document root <none> and password file <none>
I20260620 08:04:32.376188 5658 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:32.376238 5658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:32.376376 5658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:32.377449 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "afa532b5e4ef4b0cacf7c1940a3bad55"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "4464728abc5f39d001734ee3b488ed13"
server_key_iv: "ff34162151c28387dcb546ff1c2571dc"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.377905 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "afa532b5e4ef4b0cacf7c1940a3bad55"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "4464728abc5f39d001734ee3b488ed13"
server_key_iv: "ff34162151c28387dcb546ff1c2571dc"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.379295 5658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:32.380191 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.380403 5658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:32.380508 5658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "afa532b5e4ef4b0cacf7c1940a3bad55"
format_stamp: "Formatted at 2026-06-20 08:04:32 on dist-test-slave-c4w8"
server_key: "4464728abc5f39d001734ee3b488ed13"
server_key_iv: "ff34162151c28387dcb546ff1c2571dc"
server_key_version: "encryptionkey@0"
I20260620 08:04:32.380591 5658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:32.399106 5658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:32.401458 5658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:32.401713 5658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:32.401997 5658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:32.402360 5658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:32.402417 5658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.402463 5658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:32.402488 5658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:32.408542 5658 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:43691
I20260620 08:04:32.408628 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:43691 every 8 connection(s)
I20260620 08:04:32.408953 5658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:32.411556 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5658
I20260620 08:04:32.411646 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:32.411844 32207 external_mini_cluster.cc:1428] Setting key 6e4e58a0967513fa2b5964c99ea2c739
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:32.419082 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.409777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:60823 (local address 127.31.115.254:41577)
0620 08:04:32.409855 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.409857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.410065 (+ 208us) server_negotiation.cc:407] Connection header received
0620 08:04:32.411066 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.411067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.411080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.411138 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.412044 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.412681 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.413786 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.413989 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.414073 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.416278 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:32.416279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:32.416280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:32.416288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:32.417655 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.418056 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.418057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.418058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.418118 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.418392 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.418393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.418394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.418525 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:32.418600 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:32.418897 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 08:04:32.418979 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:04:32.419265 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.409763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41577 (local address 127.31.115.195:60823)
0620 08:04:32.409934 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:32.409981 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:32.410951 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:32.411216 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:32.411220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:32.411626 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:32.411961 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:32.411968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.412791 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.412793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:32.413683 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:32.413685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.414082 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.414083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:32.414229 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.414742 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:32.414751 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:32.416178 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:32.417736 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:32.417744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:32.417746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:32.417976 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:32.418192 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:32.418193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:32.418194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:32.418321 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:32.418601 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:32.418609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:32.418756 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 08:04:32.418861 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:04:32.419023 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260620 08:04:32.419806 5787 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41577
I20260620 08:04:32.419914 5787 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:32.420097 5787 heartbeater.cc:511] Master 127.31.115.254:41577 requested a full tablet report, sending...
I20260620 08:04:32.420495 5328 ts_manager.cc:194] Registered new tserver with Master: afa532b5e4ef4b0cacf7c1940a3bad55 (127.31.115.195:43691)
I20260620 08:04:32.420892 5328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:60823
I20260620 08:04:32.423589 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/test-admin.keytab.
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.439292 32207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 08:04:32.441144 5794 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:32.440193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.31.115.193:39303)
0620 08:04:32.440367 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.440372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.440383 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:32.440451 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.440454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.440474 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.440563 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.441055 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45032: BlockingRecv error: recv got EOF from 127.0.0.1:45032 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
W20260620 08:04:32.441272 32207 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc' of type FILE
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.442374 32207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.443432 32207 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:32.452382 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.444025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.31.115.193:39303)
0620 08:04:32.444092 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.444093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.444124 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:04:32.444188 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.444188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.444199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.444251 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.444692 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.445439 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.446336 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.446714 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.446799 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.449405 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:32.449417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:32.449417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:32.449425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:32.451114 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.451512 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.451516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.451517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.451581 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.451778 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.451778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.451778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.451932 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:32.452044 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:32.452225 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 08:04:32.452299 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 08:04:32.454715 5794 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:32.453959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.31.115.193:39303)
0620 08:04:32.454016 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.454019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.454066 (+ 47us) server_negotiation.cc:407] Connection header received
0620 08:04:32.454172 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.454173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.454183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.454246 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.454655 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45046: BlockingRecv error: recv got EOF from 127.0.0.1:45046 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:04:32.455014 32207 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc' of type FILE
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.456053 32207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.456920 32207 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:32.464687 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.457452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.31.115.193:39303)
0620 08:04:32.457513 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.457515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.457587 (+ 72us) server_negotiation.cc:407] Connection header received
0620 08:04:32.457757 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.457758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.457771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.457829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.458242 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.458734 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.459446 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.459657 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.459743 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.462060 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:32.462062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:32.462062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:32.462069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:32.463463 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.463854 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.463856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.463856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.463918 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.464136 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.464137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.464137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.464259 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:32.464366 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:32.464486 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 08:04:32.464572 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 08:04:32.466444 32207 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781942645567166-32207-0/krb5kdc/krb5cc' of type FILE
W20260620 08:04:32.466459 5794 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:32.465841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.31.115.193:39303)
0620 08:04:32.465898 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.465900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.465939 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:04:32.465991 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.465991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.466000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.466045 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.466406 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45062: BlockingRecv error: recv got EOF from 127.0.0.1:45062 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.467468 32207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:32.468752 32207 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:04:32.476595 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:32.469424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.31.115.193:39303)
0620 08:04:32.469460 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:32.469462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:32.469507 (+ 45us) server_negotiation.cc:407] Connection header received
0620 08:04:32.469561 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:32.469562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:32.469570 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:32.469707 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:32.470100 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.470594 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.471323 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:32.471563 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:32.471630 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:32.473952 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:32.473953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:32.473954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:32.473960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:32.475506 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.475884 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.475885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.475886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.475959 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:32.476180 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:32.476181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:32.476181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:32.476285 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:32.476360 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:32.476452 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 08:04:32.476526 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:04:32.477483 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5390
I20260620 08:04:32.482775 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5524
I20260620 08:04:32.487589 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5658
I20260620 08:04:32.492023 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5296
2026-06-20T08:04:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2005 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5807](info): set up 2 sockets
Jun 20 08:04:32 dist-test-slave-c4w8 krb5kdc[5807](info): commencing operation
krb5kdc: starting...
W20260620 08:04:33.538594 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:33 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab.
Entry for principal oryx/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab.
I20260620 08:04:33.674119 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34223
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab
--principal=oryx/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:33.760906 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:33.761129 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:33.761153 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:33.762308 5821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:33.762346 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:33.762362 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:33.762374 5821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:33.762387 5821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:33.763859 5821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:34223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab
--principal=oryx/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:33.764066 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:33.764250 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:33.769410 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:33.769420 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:33.769420 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:33.769758 5821 server_base.cc:1034] running on GCE node
I20260620 08:04:33.770037 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:33.770344 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:33.771522 5821 hybrid_clock.cc:648] HybridClock initialized: now 1781942673771489 us; error 49 us; skew 500 ppm
Jun 20 08:04:33 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:33.774326 5821 init.cc:377] Logged in from keytab as oryx/127.31.115.254@KRBTEST.COM (short username oryx)
I20260620 08:04:33.774871 5821 webserver.cc:460] Webserver started at http://127.31.115.254:39177/ using document root <none> and password file <none>
I20260620 08:04:33.775161 5821 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:33.775211 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:33.775359 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:33.776551 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "faec5a8699d8449caee9416190eaabe5"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "2f57830809c7e07d187153b6f7bb2f95"
server_key_iv: "a53fe228f9c7be0ffe4df75924d039f4"
server_key_version: "encryptionkey@0"
I20260620 08:04:33.776954 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "faec5a8699d8449caee9416190eaabe5"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "2f57830809c7e07d187153b6f7bb2f95"
server_key_iv: "a53fe228f9c7be0ffe4df75924d039f4"
server_key_version: "encryptionkey@0"
I20260620 08:04:33.778723 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 08:04:33.779654 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:33.779965 5821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:33.780082 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "faec5a8699d8449caee9416190eaabe5"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "2f57830809c7e07d187153b6f7bb2f95"
server_key_iv: "a53fe228f9c7be0ffe4df75924d039f4"
server_key_version: "encryptionkey@0"
I20260620 08:04:33.780169 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:33.795832 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:33.798244 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:33.798393 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:33.802894 5821 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:34223
I20260620 08:04:33.802977 5888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:34223 every 8 connection(s)
I20260620 08:04:33.803354 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:33.804378 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:33.806740 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Bootstrap starting.
I20260620 08:04:33.807345 5889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:33.807636 5889 log.cc:824] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:33.808368 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: No bootstrap required, opened a new log
I20260620 08:04:33.809926 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5821
I20260620 08:04:33.810061 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:33.810262 32207 external_mini_cluster.cc:1428] Setting key 057da92223edca57325b799cdd9105bf
I20260620 08:04:33.810386 5889 raft_consensus.cc:348] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } }
I20260620 08:04:33.810557 5889 raft_consensus.cc:374] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:33.810583 5889 raft_consensus.cc:729] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faec5a8699d8449caee9416190eaabe5, State: Initialized, Role: FOLLOWER
I20260620 08:04:33.810814 5889 consensus_queue.cc:260] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } }
I20260620 08:04:33.810925 5889 raft_consensus.cc:388] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:33.810971 5889 raft_consensus.cc:482] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:33.811014 5889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:33.812240 5889 raft_consensus.cc:504] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } }
I20260620 08:04:33.812354 5889 leader_election.cc:302] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faec5a8699d8449caee9416190eaabe5; no voters:
I20260620 08:04:33.812530 5889 leader_election.cc:288] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:33.812593 5894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:33.812821 5894 raft_consensus.cc:686] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 LEADER]: Becoming Leader. State: Replica: faec5a8699d8449caee9416190eaabe5, State: Running, Role: LEADER
I20260620 08:04:33.812925 5894 consensus_queue.cc:237] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } }
I20260620 08:04:33.812978 5889 sys_catalog.cc:564] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:33.813293 5896 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faec5a8699d8449caee9416190eaabe5. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } }
I20260620 08:04:33.813555 5896 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:33.813472 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } }
I20260620 08:04:33.813856 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:33.814038 5904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:33.814574 5904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:33.816313 5904 catalog_manager.cc:1347] Generated new cluster ID: 55dc5a175cee4c7b9141fd115dece84c
I20260620 08:04:33.816363 5904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:33 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:33.822499 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:33.810882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.31.115.254:34223)
0620 08:04:33.811065 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:33.811077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:33.811198 (+ 121us) server_negotiation.cc:407] Connection header received
0620 08:04:33.811335 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:33.811342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:33.811398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:33.811518 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:33.812393 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:33.814133 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:33.815321 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:33.815913 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:33.816039 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:33.818657 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:33.818662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:33.818662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:33.818678 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:33.820561 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:33.820985 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:33.820991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:33.820996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:33.821106 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:33.821341 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:33.821342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:33.821342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:33.821683 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:33.821857 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:33.822104 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 08:04:33.822278 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260620 08:04:33.836570 5904 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:33.837321 5904 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:33.845867 5904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Generated new TSK 0
I20260620 08:04:33.846101 5904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab.
Entry for principal oryx/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab.
I20260620 08:04:33.881424 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34223
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:33.966130 5916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:33.966374 5916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:33.966416 5916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:33.967588 5916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:33.967629 5916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:33.967656 5916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:33.969076 5916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:33.969270 5916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:33.969451 5916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:33.974629 5924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:33.974607 5921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:33.974607 5922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:33.975301 5916 server_base.cc:1034] running on GCE node
I20260620 08:04:33.975574 5916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:33.975822 5916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:33.977008 5916 hybrid_clock.cc:648] HybridClock initialized: now 1781942673976987 us; error 38 us; skew 500 ppm
Jun 20 08:04:33 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:33.979852 5916 init.cc:377] Logged in from keytab as oryx/127.31.115.193@KRBTEST.COM (short username oryx)
I20260620 08:04:33.980377 5916 webserver.cc:460] Webserver started at http://127.31.115.193:34643/ using document root <none> and password file <none>
I20260620 08:04:33.980623 5916 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:33.980696 5916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:33.980847 5916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:33.982026 5916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "4afebc8d250f49698ee5741ed84d845c"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "a3b9b4e4657c6bed93986bf686285104"
server_key_iv: "d2040d23c39e755196d66af6b9893d8e"
server_key_version: "encryptionkey@0"
I20260620 08:04:33.982386 5916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "4afebc8d250f49698ee5741ed84d845c"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "a3b9b4e4657c6bed93986bf686285104"
server_key_iv: "d2040d23c39e755196d66af6b9893d8e"
server_key_version: "encryptionkey@0"
I20260620 08:04:33.983845 5916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 08:04:33.984789 5931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:33.985018 5916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:33.985143 5916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "4afebc8d250f49698ee5741ed84d845c"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "a3b9b4e4657c6bed93986bf686285104"
server_key_iv: "d2040d23c39e755196d66af6b9893d8e"
server_key_version: "encryptionkey@0"
I20260620 08:04:33.985231 5916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.007201 5916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.009975 5916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:34.010177 5916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.010459 5916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:34.010812 5916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:34.010870 5916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.010916 5916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:34.010963 5916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.017980 5916 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:40327
I20260620 08:04:34.018033 6045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:40327 every 8 connection(s)
I20260620 08:04:34.018374 5916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:34.027813 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 5916
I20260620 08:04:34.027927 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:34.028205 32207 external_mini_cluster.cc:1428] Setting key 89939ece4f5641c7b9b241dcac027b2e
I20260620 08:04:34.029096 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.019194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:40829 (local address 127.31.115.254:34223)
0620 08:04:34.019268 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.019270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.019502 (+ 232us) server_negotiation.cc:407] Connection header received
0620 08:04:34.020066 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.020067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.020082 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.020146 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.021255 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.021825 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.022988 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.023203 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.023298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.025776 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.025778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.025778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.025785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.027165 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.027615 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.027616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.027617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.027701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.028196 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.028198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.028199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.028365 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.028450 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.028830 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.028956 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:34.029242 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.019138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:40829)
0620 08:04:34.019354 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.019401 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.019950 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.020234 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.020238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.020895 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.021150 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.021155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.021921 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.021922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.022900 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.022904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.023308 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.023309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.023424 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.024098 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.024107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.025682 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.027259 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.027266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.027268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.027504 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.027881 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.027956 (+ 75us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.027956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.028116 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.028466 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.028473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.028661 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.028778 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.028962 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260620 08:04:34.029887 6046 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34223
I20260620 08:04:34.030020 6046 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:34.030225 6046 heartbeater.cc:511] Master 127.31.115.254:34223 requested a full tablet report, sending...
I20260620 08:04:34.030844 5853 ts_manager.cc:194] Registered new tserver with Master: 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:34.031553 5853 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:40829
WARNING: no policy specified for oryx/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab.
Entry for principal oryx/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab.
I20260620 08:04:34.057188 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34223
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:34.143399 6051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:34.143620 6051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:34.143643 6051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:34.144845 6051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:34.144891 6051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:34.144918 6051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:34.146494 6051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:34.146775 6051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:34.147017 6051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:34.152298 6059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.152369 6056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.152562 6051 server_base.cc:1034] running on GCE node
W20260620 08:04:34.152298 6057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.152825 6051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:34.153085 6051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:34.154285 6051 hybrid_clock.cc:648] HybridClock initialized: now 1781942674154256 us; error 43 us; skew 500 ppm
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:34.156764 6051 init.cc:377] Logged in from keytab as oryx/127.31.115.194@KRBTEST.COM (short username oryx)
I20260620 08:04:34.157270 6051 webserver.cc:460] Webserver started at http://127.31.115.194:36235/ using document root <none> and password file <none>
I20260620 08:04:34.157518 6051 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.157588 6051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.157753 6051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:34.159082 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "fe4e61e11f49480db968cbb18a9b438f"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "e3d03565b2e7a7046b2db10377ece8fa"
server_key_iv: "b58a1b20a0974f5234d8e15c6b0f6d07"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.159464 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "fe4e61e11f49480db968cbb18a9b438f"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "e3d03565b2e7a7046b2db10377ece8fa"
server_key_iv: "b58a1b20a0974f5234d8e15c6b0f6d07"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.160827 6051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:34.161783 6066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.161972 6051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:34.162092 6051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "fe4e61e11f49480db968cbb18a9b438f"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "e3d03565b2e7a7046b2db10377ece8fa"
server_key_iv: "b58a1b20a0974f5234d8e15c6b0f6d07"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.162171 6051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.174757 6051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.176994 6051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:34.177141 6051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.177407 6051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:34.177812 6051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:34.177870 6051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.177913 6051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:34.177940 6051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.184053 6051 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:44645
I20260620 08:04:34.184134 6181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:44645 every 8 connection(s)
I20260620 08:04:34.184468 6051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:34.192826 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6051
I20260620 08:04:34.192935 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:34.193184 32207 external_mini_cluster.cc:1428] Setting key c9fa1f4f98cd8d2e41079b295dc6c2d0
I20260620 08:04:34.194589 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.185255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:35371 (local address 127.31.115.254:34223)
0620 08:04:34.185297 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.185301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.185574 (+ 273us) server_negotiation.cc:407] Connection header received
0620 08:04:34.186109 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.186111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.186123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.186190 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.187172 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.187746 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.188919 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.189127 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.189195 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.191504 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.191505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.191506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.191513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.193055 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.193503 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.193504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.193505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.193573 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.193872 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.193873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.193873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.193993 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.194075 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.194349 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.194440 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:04:34.194728 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.185218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:35371)
0620 08:04:34.185392 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.185436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.185997 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.186268 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.186271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.186844 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.187039 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.187044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.187857 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.187858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.188818 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.188821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.189212 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.189213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.189315 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.189966 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.189975 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.191404 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.193137 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.193143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.193145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.193415 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.193688 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.193689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.193690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.193799 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.194072 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.194080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.194216 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.194329 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.194487 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260620 08:04:34.195498 6182 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34223
I20260620 08:04:34.195667 6182 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:34.195868 6182 heartbeater.cc:511] Master 127.31.115.254:34223 requested a full tablet report, sending...
I20260620 08:04:34.196255 5853 ts_manager.cc:194] Registered new tserver with Master: fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645)
I20260620 08:04:34.196635 5853 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:35371
WARNING: no policy specified for oryx/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab.
Entry for principal oryx/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab.
I20260620 08:04:34.220549 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34223
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:34.303040 6187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:34.303313 6187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:34.303352 6187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:34.304529 6187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:34.304598 6187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:34.304643 6187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:34.306151 6187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:34.306421 6187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:34.306676 6187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:34.312090 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.312068 6195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.312064 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.312493 6187 server_base.cc:1034] running on GCE node
I20260620 08:04:34.312728 6187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:34.312984 6187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:34.314174 6187 hybrid_clock.cc:648] HybridClock initialized: now 1781942674314130 us; error 53 us; skew 500 ppm
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:34.317035 6187 init.cc:377] Logged in from keytab as oryx/127.31.115.195@KRBTEST.COM (short username oryx)
I20260620 08:04:34.317541 6187 webserver.cc:460] Webserver started at http://127.31.115.195:45545/ using document root <none> and password file <none>
I20260620 08:04:34.317826 6187 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.317898 6187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.318037 6187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:34.319182 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "71c8a779dc0b4df9af438cf8e2ca54a3"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "6ab4b30ca71f7be0ef9af88341c4bf56"
server_key_iv: "0bba6767aba65eea72a125b7ac0fbb58"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.319573 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "71c8a779dc0b4df9af438cf8e2ca54a3"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "6ab4b30ca71f7be0ef9af88341c4bf56"
server_key_iv: "0bba6767aba65eea72a125b7ac0fbb58"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.320972 6187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:34.321785 6202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.321961 6187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:34.322060 6187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "71c8a779dc0b4df9af438cf8e2ca54a3"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "6ab4b30ca71f7be0ef9af88341c4bf56"
server_key_iv: "0bba6767aba65eea72a125b7ac0fbb58"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.322151 6187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.345055 6187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.347375 6187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:34.347546 6187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.347803 6187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:34.348157 6187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:34.348214 6187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.348258 6187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:34.348285 6187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.354507 6187 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41283
I20260620 08:04:34.354573 6315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41283 every 8 connection(s)
I20260620 08:04:34.354875 6187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:34.355929 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6187
I20260620 08:04:34.356022 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:34.356206 32207 external_mini_cluster.cc:1428] Setting key 409e99268d3551cac5b0d2a96bee957c
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:34.364194 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.355599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:59427 (local address 127.31.115.254:34223)
0620 08:04:34.355663 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.355666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.355923 (+ 257us) server_negotiation.cc:407] Connection header received
0620 08:04:34.356556 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.356557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.356570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.356642 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.357720 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.358224 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.359173 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.359361 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.359426 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.361474 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.361476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.361476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.361482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.362777 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.363189 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.363190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.363191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.363255 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.363523 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.363525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.363525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.363699 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.363793 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.364049 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.364125 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:04:34.364403 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.355594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.195:59427)
0620 08:04:34.355767 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.355817 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.356478 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.356695 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.356698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.357374 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.357597 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.357644 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.358309 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.358309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.359087 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.359089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.359442 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.359443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.359542 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.360002 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.360011 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.361389 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.362873 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.362879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.362881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.363115 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.363321 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.363322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.363322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.363423 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.363776 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.363781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.363900 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.363988 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.364114 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260620 08:04:34.364908 6316 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34223
I20260620 08:04:34.365031 6316 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:34.365228 6316 heartbeater.cc:511] Master 127.31.115.254:34223 requested a full tablet report, sending...
I20260620 08:04:34.365679 5853 ts_manager.cc:194] Registered new tserver with Master: 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283)
I20260620 08:04:34.366170 5853 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:59427
I20260620 08:04:34.368422 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.115.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.253.keytab.
Entry for principal oryx/127.31.115.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.253.keytab.
I20260620 08:04:34.396422 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:35265
--webserver_interface=127.31.115.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.253.keytab
--principal=oryx/127.31.115.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:34223,127.31.115.253:35265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:34.481035 6322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:34.481310 6322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:34.481354 6322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:34.482762 6322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:34.482843 6322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:34.482867 6322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:34.482889 6322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:34.482903 6322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:34.484530 6322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:34223,127.31.115.253:35265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.253.keytab
--principal=oryx/127.31.115.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:35265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.115.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
W20260620 08:04:34.484643 6322 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.115.254:34223,127.31.115.253:35265'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 08:04:34.484766 6322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:34.484943 6322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:34.490069 6328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.490078 6327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.490090 6330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.490255 6322 server_base.cc:1034] running on GCE node
I20260620 08:04:34.490497 6322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:34.490854 6322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:34.492012 6322 hybrid_clock.cc:648] HybridClock initialized: now 1781942674491997 us; error 31 us; skew 500 ppm
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:34.494851 6322 init.cc:377] Logged in from keytab as oryx/127.31.115.253@KRBTEST.COM (short username oryx)
I20260620 08:04:34.495363 6322 webserver.cc:460] Webserver started at http://127.31.115.253:43809/ using document root <none> and password file <none>
I20260620 08:04:34.495605 6322 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.495651 6322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.495784 6322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:34.496881 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/instance:
uuid: "e869551e011b4ea59efb71ecc8d0169e"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "352a4ffab23f5d8d5560e00fd6190b82"
server_key_iv: "9ab1dacd4fe478d4d90c587e9ff6971e"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.497255 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal/instance:
uuid: "e869551e011b4ea59efb71ecc8d0169e"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "352a4ffab23f5d8d5560e00fd6190b82"
server_key_iv: "9ab1dacd4fe478d4d90c587e9ff6971e"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.498771 6322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:34.499722 6337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.499969 6322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:34.500074 6322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
uuid: "e869551e011b4ea59efb71ecc8d0169e"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "352a4ffab23f5d8d5560e00fd6190b82"
server_key_iv: "9ab1dacd4fe478d4d90c587e9ff6971e"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.500151 6322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.511235 6322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.514134 6322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:34.514312 6322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.518663 6322 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.253:35265
I20260620 08:04:34.518719 6389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.253:35265 every 8 connection(s)
I20260620 08:04:34.519129 6322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
I20260620 08:04:34.520288 6390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:34.521698 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6322
I20260620 08:04:34.522033 6390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.253@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
I20260620 08:04:34.531502 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.522840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.31.115.254:34223)
0620 08:04:34.522893 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.522896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.523035 (+ 139us) server_negotiation.cc:407] Connection header received
0620 08:04:34.523277 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.523278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.523292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.523351 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.523922 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.524446 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.525444 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.525717 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.525803 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.528212 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.528214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.528215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.528223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.529965 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.530459 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.530461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.530461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.530528 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.530782 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.530783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.530783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.530952 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.531032 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.531319 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.531404 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:04:34.531761 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.522382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.31.115.253:35265)
0620 08:04:34.522550 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.522558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.522607 (+ 49us) server_negotiation.cc:407] Connection header received
0620 08:04:34.523151 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.523159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.523181 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.523308 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.523932 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.524737 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.525520 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.525906 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.526012 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.528686 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.528694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.528694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.528701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.530369 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.530761 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.530764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.530764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.530825 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.531027 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.531028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.531028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.531203 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.531357 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.531450 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.531531 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260620 08:04:34.531798 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.522764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36258)
0620 08:04:34.522917 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.522963 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.523164 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.523372 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.523375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.523661 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.523825 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.523829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.524557 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.524558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.525347 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.525349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.525813 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.525814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.525933 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.526466 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.526476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.528094 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.530073 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.530080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.530083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.530373 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.530601 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.530602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.530602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.530715 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.531035 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.531043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.531196 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.531287 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.531450 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260620 08:04:34.532461 6390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.253@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
I20260620 08:04:34.539009 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.532765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40030)
0620 08:04:34.532799 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.532809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.532854 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.532958 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.532958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.533069 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.533102 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.533103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.533720 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.533721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.534221 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.534223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.534622 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.534624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.534697 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.535209 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.535213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.536551 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.538088 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.538089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.538090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.538258 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.538484 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.538485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.538485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.538537 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.538775 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.538776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.538812 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.538858 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.538940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 08:04:34.539083 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.532789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.31.115.253:35265)
0620 08:04:34.532839 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.532840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.532848 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:34.532899 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.532900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.532907 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.532953 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.533141 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.533647 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.534319 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.534523 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.534587 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.536706 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.536707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.536708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.536714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.538006 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.538351 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.538353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.538353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.538414 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.538599 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.538600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.538600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.538713 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.538771 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.538883 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.538993 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":31}
I20260620 08:04:34.540501 6390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Bootstrap starting.
I20260620 08:04:34.541381 6390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:34.541723 6390 log.cc:824] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:34.542483 6390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: No bootstrap required, opened a new log
I20260620 08:04:34.543938 6390 raft_consensus.cc:348] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } }
I20260620 08:04:34.544090 6390 raft_consensus.cc:374] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:34.544137 6390 raft_consensus.cc:729] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e869551e011b4ea59efb71ecc8d0169e, State: Initialized, Role: FOLLOWER
I20260620 08:04:34.544278 6390 consensus_queue.cc:260] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } }
I20260620 08:04:34.544591 6390 sys_catalog.cc:564] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:34.544544 6396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } } }
I20260620 08:04:34.544692 6396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:04:34.546481 6407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:04:34.546562 6407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:04:34.548154 6322 master_runner.cc:428] Detected that this master e869551e011b4ea59efb71ecc8d0169e is joining an existing cluster
I20260620 08:04:34.548219 6322 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.115.253:35265
I20260620 08:04:34.556457 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.549527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.31.115.254:34223)
0620 08:04:34.549609 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.549612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.549796 (+ 184us) server_negotiation.cc:407] Connection header received
0620 08:04:34.549866 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.549867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.549877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.549954 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.550348 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.551014 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.552163 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.552467 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.552551 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.553598 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.553599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.553600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.553642 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.555214 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555610 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.555612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.555612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.555703 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555881 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.555883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.555883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.556018 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.556091 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.556270 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.556351 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:04:34.556561 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.549572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.31.115.253:35265)
0620 08:04:34.549655 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.549657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.549975 (+ 318us) server_negotiation.cc:407] Connection header received
0620 08:04:34.550202 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.550203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.550213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.550278 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.550752 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.551336 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.552068 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.552359 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.552425 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.553566 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.553567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.553568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.553574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.555287 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555709 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.555710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.555711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.555762 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555927 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.555928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.555928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.556048 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.556118 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.556277 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.556427 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":65}
I20260620 08:04:34.556681 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.549412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40032)
0620 08:04:34.549911 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.549928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.549996 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.550301 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.550302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.550581 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.550664 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.550666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.551422 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.551423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.551983 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.551985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.552451 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.552453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.552522 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.553001 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.553008 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.553470 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.555376 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555376 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.555377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.555637 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.555820 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.555821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.555878 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.556140 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.556140 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.556177 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.556305 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.556573 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":182,"thread_start_us":90,"threads_started":1}
I20260620 08:04:34.556510 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.549411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36272)
0620 08:04:34.549714 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.549726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.549798 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.549960 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.549961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.550177 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.550287 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.550289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.551150 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.551151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.552077 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.552079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.552549 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.552550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.552616 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.553147 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.553152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.553488 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.555321 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.555323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.555525 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.555760 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.555760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.555761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.555822 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.556096 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.556097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.556152 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.556239 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.556359 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":81,"threads_started":1}
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.254@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
I20260620 08:04:34.567541 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.559483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40042)
0620 08:04:34.559667 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.559710 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.559805 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.559951 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.559955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.560248 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.560308 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.560311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.560931 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.560932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.561543 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.561545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.561962 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.561963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.562052 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.562538 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.562543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.564040 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.566111 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.566125 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.566128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.566446 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.566758 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.566759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.566759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.566830 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.567174 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.567183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.567245 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.567334 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.567462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260620 08:04:34.567564 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.559559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.31.115.253:35265)
0620 08:04:34.559600 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.559602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.559777 (+ 175us) server_negotiation.cc:407] Connection header received
0620 08:04:34.559873 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.559874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.559885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.559928 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.560387 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.560848 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.561647 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.561870 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.561970 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.564204 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.564206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.564206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.564214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.565961 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.566593 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.566594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.566595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.566659 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.566915 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.566916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.566916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.567068 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.567144 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.567377 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.567464 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:34.568536 5853 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.115.253:35265: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e"
member_type: NON_VOTER
last_known_addr {
host: "127.31.115.253"
port: 35265
}
attrs {
promote: true
}
}
dest_uuid: "faec5a8699d8449caee9416190eaabe5"
cas_config_opid_index: -1
I20260620 08:04:34.568884 5853 consensus_queue.cc:237] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } }
I20260620 08:04:34.569408 6417 raft_consensus.cc:2932] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e869551e011b4ea59efb71ecc8d0169e (127.31.115.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } }
I20260620 08:04:34.569802 6364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:34.570633 6417 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.115.253:35265
I20260620 08:04:34.570737 6364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Refusing update from remote peer faec5a8699d8449caee9416190eaabe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 08:04:34.570896 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } }
I20260620 08:04:34.570989 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:34.571153 6322 master.cc:537] Master@127.31.115.253:35265 shutting down...
I20260620 08:04:34.571137 6416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:04:34.572535 5837 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 08:04:34.573216 5837 consensus_peers.cc:487] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 -> Peer e869551e011b4ea59efb71ecc8d0169e (127.31.115.253:35265): Couldn't send request to peer e869551e011b4ea59efb71ecc8d0169e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:04:34.574092 6322 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:34.574191 6322 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:34.574242 6322 tablet_replica.cc:330] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: stopping tablet replica
I20260620 08:04:34.587845 6322 master.cc:552] Master@127.31.115.253:35265 shutdown complete.
I20260620 08:04:34.587913 6322 master_runner.cc:305] Clearing existing system tablet
I20260620 08:04:34.588050 6322 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.588109 6322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.588913 6322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:34.589826 6421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.589998 6322 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.590072 6322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:04:34.590099 6322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
uuid: "e869551e011b4ea59efb71ecc8d0169e"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "352a4ffab23f5d8d5560e00fd6190b82"
server_key_iv: "9ab1dacd4fe478d4d90c587e9ff6971e"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.590982 6322 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:34.592605 6322 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 08:04:34.592689 6322 log.cc:1196] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 08:04:34.592888 6322 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Deleting consensus metadata
I20260620 08:04:34.593014 6322 master_runner.cc:315] Copying system tablet from 127.31.115.254:34223
I20260620 08:04:34.593677 6322 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: tablet copy: Beginning tablet copy session from remote peer at address 127.31.115.254:34223
I20260620 08:04:34.601055 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.594025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36286)
0620 08:04:34.594187 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.594200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.594277 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.594406 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.594406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.594601 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.594664 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.594666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.595359 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.595360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.596257 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.596261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.596683 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.596684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.596755 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.597539 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.597545 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.598040 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.599782 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.599783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.599784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.600014 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.600251 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.600252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.600253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.600322 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.600698 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.600699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.600749 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.600823 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.600932 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260620 08:04:34.601070 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.594120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.31.115.254:34223)
0620 08:04:34.594186 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.594189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.594290 (+ 101us) server_negotiation.cc:407] Connection header received
0620 08:04:34.594329 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.594330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.594342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.594404 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.594739 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.595273 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.596360 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.596598 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.596668 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.598151 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.598153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.598153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.598160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.599694 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.600101 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.600102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.600102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.600187 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.600438 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.600440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.600440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.600613 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.600692 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.600920 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.600999 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:04:34.601461 5873 tablet_copy_service.cc:140] P faec5a8699d8449caee9416190eaabe5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e869551e011b4ea59efb71ecc8d0169e ({username='oryx', principal='oryx/127.31.115.253@KRBTEST.COM'} at 127.0.0.1:36286)
I20260620 08:04:34.601573 5873 tablet_copy_service.cc:161] P faec5a8699d8449caee9416190eaabe5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e869551e011b4ea59efb71ecc8d0169e at {username='oryx', principal='oryx/127.31.115.253@KRBTEST.COM'} at 127.0.0.1:36286: session id = e869551e011b4ea59efb71ecc8d0169e-00000000000000000000000000000000
I20260620 08:04:34.602437 5873 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 08:04:34.603113 6322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 08:04:34.604226 6322 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: tablet copy: Starting download of 0 data blocks...
I20260620 08:04:34.604408 6322 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: tablet copy: Starting download of 1 WAL segments...
I20260620 08:04:34.605508 6322 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 08:04:34.606427 5873 tablet_copy_service.cc:342] P faec5a8699d8449caee9416190eaabe5: Request end of tablet copy session e869551e011b4ea59efb71ecc8d0169e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.115.253@KRBTEST.COM'} at 127.0.0.1:36286
I20260620 08:04:34.606489 5873 tablet_copy_service.cc:433] P faec5a8699d8449caee9416190eaabe5: ending tablet copy session e869551e011b4ea59efb71ecc8d0169e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e869551e011b4ea59efb71ecc8d0169e
I20260620 08:04:34.608893 6322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:34.610328 6429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.610476 6322 server_base.cc:1034] running on GCE node
W20260620 08:04:34.610596 6431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.610337 6428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.610857 6322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:34.611035 6322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:34.612177 6322 hybrid_clock.cc:648] HybridClock initialized: now 1781942674612168 us; error 24 us; skew 500 ppm
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:34.613829 6322 init.cc:377] Logged in from keytab as oryx/127.31.115.253@KRBTEST.COM (short username oryx)
I20260620 08:04:34.614116 6322 webserver.cc:460] Webserver started at http://127.31.115.253:34557/ using document root <none> and password file <none>
I20260620 08:04:34.614264 6322 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.614298 6322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.614923 6322 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.615483 6438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.615608 6322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.615655 6322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
uuid: "e869551e011b4ea59efb71ecc8d0169e"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "352a4ffab23f5d8d5560e00fd6190b82"
server_key_iv: "9ab1dacd4fe478d4d90c587e9ff6971e"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.615715 6322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.622726 6322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.624428 6322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.627640 6322 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.253:35265
I20260620 08:04:34.627954 6491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.253:35265 every 8 connection(s)
I20260620 08:04:34.628108 6322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
I20260620 08:04:34.628990 6492 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:04:34.629505 6492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Bootstrap starting.
I20260620 08:04:34.632386 6492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:04:34.632634 6492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Bootstrap complete.
I20260620 08:04:34.632853 6492 raft_consensus.cc:348] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } }
I20260620 08:04:34.632966 6492 raft_consensus.cc:729] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e869551e011b4ea59efb71ecc8d0169e, State: Initialized, Role: LEARNER
I20260620 08:04:34.633006 6492 consensus_queue.cc:260] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } }
I20260620 08:04:34.633249 6492 sys_catalog.cc:564] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:34.633241 6495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } }
I20260620 08:04:34.633359 6495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: LEARNER
I20260620 08:04:34.635186 6507 catalog_manager.cc:1259] Loaded cluster ID: 55dc5a175cee4c7b9141fd115dece84c
I20260620 08:04:34.635288 6507 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: loading cluster ID for follower catalog manager: success
I20260620 08:04:34.636502 6507 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: acquiring CA information for follower catalog manager: success
I20260620 08:04:34.636793 6507 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:04:34.641841 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.633841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.31.115.253:35265)
0620 08:04:34.633982 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.633984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.633993 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:34.634020 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.634021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.634039 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.634768 (+ 729us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.634806 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.636021 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.637014 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.637212 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.637263 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.638759 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.638760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.638760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.638767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.640347 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.640746 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.640747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.640747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.640886 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.641164 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.641165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.641165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.641278 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.641358 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.641526 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.641687 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.31.115.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.252.keytab.
Entry for principal oryx/127.31.115.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.252.keytab.
I20260620 08:04:34.670372 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:44943
--webserver_interface=127.31.115.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.252.keytab
--principal=oryx/127.31.115.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:34.760427 6510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:34.760661 6510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:34.760686 6510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:34.761986 6510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:34.762033 6510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:34.762048 6510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:34.762061 6510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:34.762074 6510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:34.763680 6510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.252.keytab
--principal=oryx/127.31.115.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.115.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:34.763964 6510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:34.764233 6510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:34.770215 6515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.770272 6518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.770253 6510 server_base.cc:1034] running on GCE node
W20260620 08:04:34.770313 6516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.770709 6510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:34.771001 6510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:34.772154 6510 hybrid_clock.cc:648] HybridClock initialized: now 1781942674772129 us; error 38 us; skew 500 ppm
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:34.775028 6510 init.cc:377] Logged in from keytab as oryx/127.31.115.252@KRBTEST.COM (short username oryx)
I20260620 08:04:34.775648 6510 webserver.cc:460] Webserver started at http://127.31.115.252:42029/ using document root <none> and password file <none>
I20260620 08:04:34.775913 6510 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.775990 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.776160 6510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:34.777379 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/instance:
uuid: "9cb7362dc7c4489aac838c9083262776"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "2d6ac689c8a4a2e6bd6e70e5435197bd"
server_key_iv: "c3069699e09c867833a53e28455b1cda"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.777812 6510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal/instance:
uuid: "9cb7362dc7c4489aac838c9083262776"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "2d6ac689c8a4a2e6bd6e70e5435197bd"
server_key_iv: "c3069699e09c867833a53e28455b1cda"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.779297 6510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:34.780146 6525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.780351 6510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:34.780472 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
uuid: "9cb7362dc7c4489aac838c9083262776"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "2d6ac689c8a4a2e6bd6e70e5435197bd"
server_key_iv: "c3069699e09c867833a53e28455b1cda"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.780552 6510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.799780 6510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.803256 6510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:34.803435 6510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.807511 6510 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.252:44943
I20260620 08:04:34.807629 6578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.252:44943 every 8 connection(s)
I20260620 08:04:34.807957 6510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
I20260620 08:04:34.809051 6579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:34.810738 6579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:04:34.816895 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6510
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.252@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:34.821719 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.811639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.31.115.254:34223)
0620 08:04:34.811751 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.811753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.811948 (+ 195us) server_negotiation.cc:407] Connection header received
0620 08:04:34.812489 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.812490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.812511 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.812590 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.813509 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.814146 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.815355 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.815661 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.815765 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.818285 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.818287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.818287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.818297 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.820101 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.820639 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.820641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.820641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.820725 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.821007 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.821008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.821009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.821134 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.821209 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.821498 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.821588 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 08:04:34.821933 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.811537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36296)
0620 08:04:34.811790 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.811837 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.812378 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.812713 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.812718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.813074 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.813384 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.813390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.814251 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.814252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.815272 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.815275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.815795 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.815796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.815940 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.816455 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.816464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.818192 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.820217 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.820224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.820226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.820493 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.820808 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.820809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.820810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.820926 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.821210 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.821217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.821358 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.821474 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.821675 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260620 08:04:34.822494 6579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.115.252@KRBTEST.COM
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.252@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
I20260620 08:04:34.827102 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.817466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.31.115.252:44943)
0620 08:04:34.817676 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.817682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.817708 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:04:34.817728 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.817729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.817748 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.817830 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.818450 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.819530 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.820530 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.820906 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.821007 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.823963 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.823972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.823973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.823980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.825774 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.826155 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.826158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.826158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.826223 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.826429 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.826430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.826430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.826580 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.826754 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.826879 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.826961 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260620 08:04:34.829759 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.822795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40074)
0620 08:04:34.822858 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.822870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.822918 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.823109 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.823110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.823243 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.823293 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.823295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.824044 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.824044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.824909 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.824912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.825262 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.825263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.825336 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.825893 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.825905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.827137 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.828708 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.828709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.828709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.828879 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.829055 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.829056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.829057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.829134 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.829367 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.829368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.829410 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.829514 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.829596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:04:34.829831 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.822903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.31.115.253:35265)
0620 08:04:34.822951 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.822952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.822961 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:34.822985 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.822986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.822996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.823054 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.823387 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.823968 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.824979 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.825188 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.825241 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.827244 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.827246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.827246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.827255 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.828607 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.828946 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.828947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.828947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.828997 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.829188 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.829188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.829189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.829284 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.829334 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.829586 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.829759 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:34.830246 6579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.252@KRBTEST.COM for oryx/127.31.115.252@KRBTEST.COM
I20260620 08:04:34.837378 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.830576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43162)
0620 08:04:34.830636 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.830648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.830692 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.830841 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.830841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.830951 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.830984 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.830985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.831559 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.831560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.832070 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.832072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.832527 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.832528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.832601 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.833066 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.833072 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.834403 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.836490 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.836491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.836491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.836670 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.836834 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.836834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.836835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.836879 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.837089 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.837090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.837132 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.837198 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.837282 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 08:04:34.837426 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.830698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.31.115.252:44943)
0620 08:04:34.830747 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.830749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.830757 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:34.830775 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.830776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.830786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.830839 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.831025 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.831473 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.832189 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.832387 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.832453 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.834505 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.834507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.834507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.834559 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.836396 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.836742 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.836743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.836743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.836791 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.836913 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.836913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.836913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.837004 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.837049 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.837229 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.837326 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":30}
I20260620 08:04:34.839161 6579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Bootstrap starting.
I20260620 08:04:34.840036 6579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:34.840338 6579 log.cc:824] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:34.841109 6579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: No bootstrap required, opened a new log
I20260620 08:04:34.842715 6579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } }
I20260620 08:04:34.842839 6579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:34.842861 6579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb7362dc7c4489aac838c9083262776, State: Initialized, Role: FOLLOWER
I20260620 08:04:34.843007 6579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } }
I20260620 08:04:34.843346 6579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:34.843326 6584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } } }
I20260620 08:04:34.843497 6584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:04:34.845271 6595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:04:34.845332 6595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:04:34.847457 6510 master_runner.cc:428] Detected that this master 9cb7362dc7c4489aac838c9083262776 is joining an existing cluster
I20260620 08:04:34.847512 6510 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.115.252:44943
I20260620 08:04:34.856503 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.848580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.31.115.254:34223)
0620 08:04:34.848631 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.848634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.849032 (+ 398us) server_negotiation.cc:407] Connection header received
0620 08:04:34.849131 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.849132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.849144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.849200 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.849769 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.850560 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.851417 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.851754 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.851828 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.852978 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.852980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.852980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.852996 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.855208 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855645 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.855646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.855647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.855708 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855921 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.855921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.855922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.856068 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.856135 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.856306 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.856394 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:34.856505 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.848680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36300)
0620 08:04:34.848951 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.848959 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.849064 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.849215 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.849216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.849538 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.849688 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.849689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.850651 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.850652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.851306 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.851308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.851852 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.851853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.851928 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.852449 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.852455 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.852875 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.855308 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.855310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.855535 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.855779 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.855780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.855848 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.856139 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.856140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.856191 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.856265 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.856371 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":18,"thread_start_us":80,"threads_started":1}
I20260620 08:04:34.856621 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.848575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.31.115.253:35265)
0620 08:04:34.848622 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.848625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.849020 (+ 395us) server_negotiation.cc:407] Connection header received
0620 08:04:34.849075 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.849076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.849089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.849141 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.850675 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.851413 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.852285 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.852539 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.852629 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.853588 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.853590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.853590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.853599 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.855326 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855751 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.855752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.855753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.855806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.856057 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.856059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.856059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.856187 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.856231 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.856453 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.856531 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:04:34.856889 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.848550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40088)
0620 08:04:34.848928 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.848941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.849013 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.849216 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.849216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.849541 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.849683 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.849686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.851496 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.851497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.852201 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.852204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.852632 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.852633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.852699 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.853149 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.853154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.853497 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.855407 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.855408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.855658 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.855877 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.855877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.855878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.855971 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.856297 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.856298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.856337 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.856715 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.856819 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942673, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.254@KRBTEST.COM for oryx/127.31.115.252@KRBTEST.COM
I20260620 08:04:34.878786 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5916
I20260620 08:04:34.883054 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.859642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43178)
0620 08:04:34.859689 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.859699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.859741 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.860118 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.860120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.860287 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.860334 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.860336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.860995 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.860996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.861848 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.861852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.862239 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.862240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.862311 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.862758 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.862763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.864134 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.881903 (+ 17769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.881904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.881905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.882161 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.882383 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.882383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.882384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.882441 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.882718 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.882719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.882770 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.882815 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.882941 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 08:04:34.883438 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.859796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.31.115.252:44943)
0620 08:04:34.859971 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.859973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.859979 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:04:34.860013 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.860014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.860023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.860085 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.860410 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.860919 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.861945 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.862164 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.862220 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.864266 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.864268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.864270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.864282 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.881780 (+ 17498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.882251 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.882253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.882253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.882310 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.882507 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.882508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.882508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.882642 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.883182 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.883244 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.883328 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":120,"threads_started":1}
I20260620 08:04:34.884307 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.848829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.31.115.252:44943)
0620 08:04:34.848869 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.848871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.857501 (+ 8630us) server_negotiation.cc:407] Connection header received
0620 08:04:34.857702 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.857703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.857712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.857739 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.858124 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.858754 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.859918 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.860207 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.860283 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.879902 (+ 19619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.879903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.879904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.879911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.882348 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.882725 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.882726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.882727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.882791 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.883030 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.883031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.883031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.883174 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.883232 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.883989 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.884177 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":20}
I20260620 08:04:34.884313 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.848807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43164)
0620 08:04:34.857338 (+ 8531us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.857350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.857599 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.857789 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.857790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.857960 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.858014 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.858016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.859003 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.859004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.859833 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.859836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.860280 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.860281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.860343 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.878534 (+ 18191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.878540 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.879756 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.882443 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.882444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.882445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.882657 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.882874 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.882875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.882875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.882931 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.883793 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.883794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.883842 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.883956 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.884116 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8519,"thread_start_us":74,"threads_started":1}
I20260620 08:04:34.886261 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:40327
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:34223
--webserver_port=34643
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:04:34.886327 5853 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.115.252:44943: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9cb7362dc7c4489aac838c9083262776"
member_type: NON_VOTER
last_known_addr {
host: "127.31.115.252"
port: 44943
}
attrs {
promote: true
}
}
dest_uuid: "faec5a8699d8449caee9416190eaabe5"
cas_config_opid_index: 5
I20260620 08:04:34.886590 5853 consensus_queue.cc:237] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } }
I20260620 08:04:34.887064 6605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9cb7362dc7c4489aac838c9083262776 (127.31.115.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.887383 6552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:34.888214 6605 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.115.252:44943
I20260620 08:04:34.888338 6552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Refusing update from remote peer faec5a8699d8449caee9416190eaabe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 08:04:34.888387 6417 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.888464 6417 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:34.888589 6510 master.cc:537] Master@127.31.115.252:44943 shutting down...
I20260620 08:04:34.888746 6605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:04:34.889429 5840 consensus_peers.cc:487] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 -> Peer 9cb7362dc7c4489aac838c9083262776 (127.31.115.252:44943): Couldn't send request to peer 9cb7362dc7c4489aac838c9083262776. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:04:34.892241 6510 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:34.892354 6510 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:34.892407 6510 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: stopping tablet replica
I20260620 08:04:34.892544 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.888049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40090)
0620 08:04:34.888151 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.888164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.888207 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.888619 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.888620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.888622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:34.888678 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.888681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.890020 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.890022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.891254 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.891259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.892259 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.892260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.892337 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.892365 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.892419 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.892465 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 08:04:34.892828 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.888288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.31.115.253:35265)
0620 08:04:34.888333 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.888336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.888344 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:34.888371 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.888373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.888399 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.888457 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:34.888822 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.889591 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.891354 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.891961 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.892037 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.892142 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.892712 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.892759 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:04:34.893224 6465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 LEARNER]: Refusing update from remote peer faec5a8699d8449caee9416190eaabe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 08:04:34.893483 6605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 08:04:34.894161 6465 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9cb7362dc7c4489aac838c9083262776 (127.31.115.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.896204 6495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: New leader faec5a8699d8449caee9416190eaabe5. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.896302 6495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: LEARNER
I20260620 08:04:34.896341 6495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: NON_VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: true } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.896381 6495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: LEARNER
I20260620 08:04:34.896723 6417 raft_consensus.cc:1050] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: attempting to promote NON_VOTER e869551e011b4ea59efb71ecc8d0169e to VOTER
I20260620 08:04:34.896936 6417 consensus_queue.cc:237] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } }
I20260620 08:04:34.897938 6465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 LEARNER]: Refusing update from remote peer faec5a8699d8449caee9416190eaabe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 08:04:34.898190 6416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 08:04:34.898728 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.897490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43184)
0620 08:04:34.898636 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.898694 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1122}
I20260620 08:04:34.899271 6416 raft_consensus.cc:2932] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e869551e011b4ea59efb71ecc8d0169e (127.31.115.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.900300 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.900357 6609 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.900492 6609 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:34.900373 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:34.900918 6465 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e869551e011b4ea59efb71ecc8d0169e (127.31.115.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.901819 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.899001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43194)
0620 08:04:34.901740 (+ 2739us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.901783 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2716}
W20260620 08:04:34.901971 5840 consensus_peers.cc:487] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 -> Peer 9cb7362dc7c4489aac838c9083262776 (127.31.115.252:44943): Couldn't send request to peer 9cb7362dc7c4489aac838c9083262776. Status: Network error: Client connection negotiation failed: client connection to 127.31.115.252:44943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 08:04:34.902192 6608 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.902292 6608 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:04:34.907251 6510 master.cc:552] Master@127.31.115.252:44943 shutdown complete.
I20260620 08:04:34.907315 6510 master_runner.cc:305] Clearing existing system tablet
I20260620 08:04:34.907454 6510 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.907505 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.908352 6510 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:34.908933 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.909106 6510 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.909169 6510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.909195 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
uuid: "9cb7362dc7c4489aac838c9083262776"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "2d6ac689c8a4a2e6bd6e70e5435197bd"
server_key_iv: "c3069699e09c867833a53e28455b1cda"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.910130 6510 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:34.911603 6510 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 08:04:34.911681 6510 log.cc:1196] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 08:04:34.911885 6510 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Deleting consensus metadata
I20260620 08:04:34.912137 6510 master_runner.cc:315] Copying system tablet from 127.31.115.254:34223
I20260620 08:04:34.912802 6510 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: tablet copy: Beginning tablet copy session from remote peer at address 127.31.115.254:34223
I20260620 08:04:34.921005 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.913164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36314)
0620 08:04:34.913290 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:34.913301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:34.913431 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:34.913664 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:34.913665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:34.913874 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:34.913937 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.913939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.914903 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.914905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.915845 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:34.915849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.916349 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.916350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:34.916425 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.917465 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:34.917471 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:34.917960 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:34.919841 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.919842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.919843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.920086 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.920301 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:34.920302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:34.920302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:34.920370 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:34.920639 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:34.920640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:34.920695 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:04:34.920776 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:04:34.920889 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260620 08:04:34.921005 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:34.913181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.31.115.254:34223)
0620 08:04:34.913263 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:34.913265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:34.913392 (+ 127us) server_negotiation.cc:407] Connection header received
0620 08:04:34.913510 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:34.913511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:34.913541 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:34.913655 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:34.914044 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.914821 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.915960 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:34.916265 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:34.916341 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:34.918072 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:34.918074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:34.918074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:34.918083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:34.919746 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.920169 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.920170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.920170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.920231 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:34.920435 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:34.920436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:34.920436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:34.920573 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:34.920639 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:34.920817 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 08:04:34.920906 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:04:34.921801 5873 tablet_copy_service.cc:140] P faec5a8699d8449caee9416190eaabe5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9cb7362dc7c4489aac838c9083262776 ({username='oryx', principal='oryx/127.31.115.252@KRBTEST.COM'} at 127.0.0.1:36314)
I20260620 08:04:34.921870 5873 tablet_copy_service.cc:161] P faec5a8699d8449caee9416190eaabe5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9cb7362dc7c4489aac838c9083262776 at {username='oryx', principal='oryx/127.31.115.252@KRBTEST.COM'} at 127.0.0.1:36314: session id = 9cb7362dc7c4489aac838c9083262776-00000000000000000000000000000000
I20260620 08:04:34.922508 5873 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 08:04:34.923048 6510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 08:04:34.924419 6510 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: tablet copy: Starting download of 0 data blocks...
I20260620 08:04:34.924556 6510 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: tablet copy: Starting download of 1 WAL segments...
I20260620 08:04:34.925449 6510 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 08:04:34.926445 5873 tablet_copy_service.cc:342] P faec5a8699d8449caee9416190eaabe5: Request end of tablet copy session 9cb7362dc7c4489aac838c9083262776-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.115.252@KRBTEST.COM'} at 127.0.0.1:36314
I20260620 08:04:34.926507 5873 tablet_copy_service.cc:433] P faec5a8699d8449caee9416190eaabe5: ending tablet copy session 9cb7362dc7c4489aac838c9083262776-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9cb7362dc7c4489aac838c9083262776
I20260620 08:04:34.928900 6510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:34.930183 6618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:34.930670 6621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.930804 6510 server_base.cc:1034] running on GCE node
W20260620 08:04:34.945344 6619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:34.945561 6510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:34.945746 6510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:34.946895 6510 hybrid_clock.cc:648] HybridClock initialized: now 1781942674946870 us; error 52 us; skew 500 ppm
Jun 20 08:04:34 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942674, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:34.948554 6510 init.cc:377] Logged in from keytab as oryx/127.31.115.252@KRBTEST.COM (short username oryx)
I20260620 08:04:34.948854 6510 webserver.cc:460] Webserver started at http://127.31.115.252:44861/ using document root <none> and password file <none>
I20260620 08:04:34.948987 6510 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:34.949026 6510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:34.949839 6510 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:34.950479 6628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:34.950634 6510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:34.950690 6510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
uuid: "9cb7362dc7c4489aac838c9083262776"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "2d6ac689c8a4a2e6bd6e70e5435197bd"
server_key_iv: "c3069699e09c867833a53e28455b1cda"
server_key_version: "encryptionkey@0"
I20260620 08:04:34.950745 6510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:34.970216 6510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:34.972167 6510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:34.975402 6510 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.252:44943
I20260620 08:04:34.975440 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.252:44943 every 8 connection(s)
I20260620 08:04:34.975862 6510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
I20260620 08:04:34.976828 6681 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:04:34.977461 6681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Bootstrap starting.
I20260620 08:04:34.980757 6681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:04:34.981006 6681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: Bootstrap complete.
I20260620 08:04:34.981230 6681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } }
I20260620 08:04:34.981321 6681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cb7362dc7c4489aac838c9083262776, State: Initialized, Role: LEARNER
I20260620 08:04:34.981360 6681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } }
I20260620 08:04:34.981545 6681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:34.981572 6684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:34.981818 6684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:04:34.984078 6695 catalog_manager.cc:1259] Loaded cluster ID: 55dc5a175cee4c7b9141fd115dece84c
I20260620 08:04:34.984139 6695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: loading cluster ID for follower catalog manager: success
I20260620 08:04:34.985173 6695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: acquiring CA information for follower catalog manager: success
I20260620 08:04:34.985496 6695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 08:04:34.990808 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:34.991050 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:34.991089 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:34.992327 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:34.992393 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:34.992439 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:34.994120 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:40327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=34643
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:34.994408 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:34.994654 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:35.000885 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:35.001024 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:35.000885 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:35.001266 6606 server_base.cc:1034] running on GCE node
I20260620 08:04:35.001660 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:35.001989 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:35.003244 6606 hybrid_clock.cc:648] HybridClock initialized: now 1781942675003219 us; error 38 us; skew 500 ppm
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:35.006701 6606 init.cc:377] Logged in from keytab as oryx/127.31.115.193@KRBTEST.COM (short username oryx)
I20260620 08:04:35.007223 6606 webserver.cc:460] Webserver started at http://127.31.115.193:34643/ using document root <none> and password file <none>
I20260620 08:04:35.007476 6606 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:35.007566 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:35.009140 6606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:35.009912 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.010272 6606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:35.010399 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "4afebc8d250f49698ee5741ed84d845c"
format_stamp: "Formatted at 2026-06-20 08:04:33 on dist-test-slave-c4w8"
server_key: "a3b9b4e4657c6bed93986bf686285104"
server_key_iv: "d2040d23c39e755196d66af6b9893d8e"
server_key_version: "encryptionkey@0"
I20260620 08:04:35.010854 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:35.022388 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:35.025156 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:35.025344 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:35.025674 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:35.026026 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:35.026084 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.026126 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:35.026155 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.032033 6606 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:40327
I20260620 08:04:35.032114 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:40327 every 8 connection(s)
I20260620 08:04:35.032472 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:35.035722 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6606
I20260620 08:04:35.035873 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6051
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.252@KRBTEST.COM
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
I20260620 08:04:35.045341 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:44645
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:34223
--webserver_port=36235
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:04:35.045897 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.033710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:59061 (local address 127.31.115.252:44943)
0620 08:04:35.034692 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.034694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.034701 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:35.034727 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.034728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.034741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.034800 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.035692 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.036805 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038223 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.038535 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038594 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.041835 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.041836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.041837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.041845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.044088 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.044712 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.044713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.044713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.044779 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.045093 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.045095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.045095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.045233 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.045289 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.045688 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.045791 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":81,"threads_started":1}
I20260620 08:04:35.046025 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.033274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:54243 (local address 127.31.115.254:34223)
0620 08:04:35.033337 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.033340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.033874 (+ 534us) server_negotiation.cc:407] Connection header received
0620 08:04:35.034720 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.034721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.034732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.034793 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.035691 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.036483 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038232 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.038539 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038582 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.042332 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.042334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.042334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.042344 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.044273 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.044735 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.044736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.044736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.044789 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.045101 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.045102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.045103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.045232 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.045288 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.045832 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.045922 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:04:35.046465 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.033600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.31.115.193:59061)
0620 08:04:35.033975 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.033986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.034424 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.034853 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.034854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.035388 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.035571 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.035577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.036900 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.036901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.038124 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.038127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.038660 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.038841 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.039562 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.039573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.041737 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.044189 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.044197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.044200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.044626 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.044853 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.044854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.044854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.045018 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.045318 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.045324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.045513 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.046023 (+ 510us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.046237 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":12,"thread_start_us":123,"threads_started":1}
I20260620 08:04:35.046584 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.033854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:41923 (local address 127.31.115.253:35265)
0620 08:04:35.033911 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.033914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.033923 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:35.034582 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.034583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.034596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.034649 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.035701 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.036487 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038230 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.038544 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038587 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.043462 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.043464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.043464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.043472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.045290 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.045708 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.045710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.045711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.045768 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.045958 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.045959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.045959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.046112 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.046163 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.046354 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.046443 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:04:35.046921 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.033254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.31.115.193:41923)
0620 08:04:35.033429 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.033473 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.034423 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.034774 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.034778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.035324 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.035571 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.035585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.036627 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.036628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.038125 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.038128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.038669 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.038840 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.039665 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.039670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.043358 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.045389 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.045390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.045391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.045624 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.045835 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.045835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.045836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.045896 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.046196 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.046197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.046244 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.046681 (+ 437us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.046801 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":74,"thread_start_us":117,"threads_started":1}
I20260620 08:04:35.046957 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.033254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:54243)
0620 08:04:35.033810 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.033821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.034425 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.034829 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.034831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.035389 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.035571 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.035577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.036708 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.036709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.038124 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.038127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.038664 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.038664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.038841 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.039588 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.039593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.042227 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.044354 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.044355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.044355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.044648 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.044855 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.044856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.044856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.045018 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.045319 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.045324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.045710 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.046463 (+ 753us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.046855 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":111,"thread_start_us":96,"threads_started":1}
I20260620 08:04:35.047356 6825 heartbeater.cc:347] Connected to a master server at 127.31.115.252:44943
I20260620 08:04:35.047475 6825 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.047503 6826 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34223
I20260620 08:04:35.047565 6826 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.047551 6827 heartbeater.cc:347] Connected to a master server at 127.31.115.253:35265
I20260620 08:04:35.047641 6827 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.047695 6825 heartbeater.cc:511] Master 127.31.115.252:44943 requested a full tablet report, sending...
I20260620 08:04:35.047698 6826 heartbeater.cc:511] Master 127.31.115.254:34223 requested a full tablet report, sending...
I20260620 08:04:35.047694 6827 heartbeater.cc:511] Master 127.31.115.253:35265 requested a full tablet report, sending...
I20260620 08:04:35.048234 5853 ts_manager.cc:194] Re-registered known tserver with Master: 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:35.048250 6455 ts_manager.cc:194] Registered new tserver with Master: 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:35.048298 6645 ts_manager.cc:194] Registered new tserver with Master: 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:35.048768 5853 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:54243
W20260620 08:04:35.133231 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:35.133455 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:35.133482 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:35.134877 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:35.134964 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:35.135003 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:35.136613 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:44645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=36235
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:35.136922 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:35.137198 6834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:35.142527 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:35.142558 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:35.142519 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:35.142903 6834 server_base.cc:1034] running on GCE node
I20260620 08:04:35.143204 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:35.143502 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:35.144745 6834 hybrid_clock.cc:648] HybridClock initialized: now 1781942675144760 us; error 65 us; skew 500 ppm
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:35.147506 6834 init.cc:377] Logged in from keytab as oryx/127.31.115.194@KRBTEST.COM (short username oryx)
I20260620 08:04:35.147994 6834 webserver.cc:460] Webserver started at http://127.31.115.194:36235/ using document root <none> and password file <none>
I20260620 08:04:35.148247 6834 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:35.148307 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:35.149863 6834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:35.150658 6849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.150959 6834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:35.151139 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "fe4e61e11f49480db968cbb18a9b438f"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "e3d03565b2e7a7046b2db10377ece8fa"
server_key_iv: "b58a1b20a0974f5234d8e15c6b0f6d07"
server_key_version: "encryptionkey@0"
I20260620 08:04:35.151566 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:35.178604 6834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:35.181036 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:35.181238 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:35.181531 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:35.181941 6834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:35.182029 6834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.182082 6834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:35.182114 6834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.188443 6834 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:44645
I20260620 08:04:35.188508 6962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:44645 every 8 connection(s)
I20260620 08:04:35.188890 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:35.192754 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6834
I20260620 08:04:35.192909 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6187
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.252@KRBTEST.COM
I20260620 08:04:35.200305 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:41283
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:34223
--webserver_port=45545
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:04:35.203615 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.189765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:44889 (local address 127.31.115.254:34223)
0620 08:04:35.189880 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.189882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.190147 (+ 265us) server_negotiation.cc:407] Connection header received
0620 08:04:35.191326 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.191328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.191343 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.191398 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.192262 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.193037 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.194729 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.195044 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.195087 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.199312 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.199314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.199315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.199330 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.201501 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202192 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.202194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.202194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.202270 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202648 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.202650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.202650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.202836 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.202906 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.203388 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.203498 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 08:04:35.203845 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.189823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:45347 (local address 127.31.115.253:35265)
0620 08:04:35.189868 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.189870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.190133 (+ 263us) server_negotiation.cc:407] Connection header received
0620 08:04:35.191511 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.191512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.191523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.191594 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.192251 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.192939 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.194725 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.195062 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.195104 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.199845 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.199846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.199847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.199855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.202309 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202912 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.202913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.202914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.202979 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.203188 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.203189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.203189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.203362 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.203416 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.203610 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.203701 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:04:35.204296 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.189705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.31.115.194:45347)
0620 08:04:35.189992 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.190038 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.191173 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.191600 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.191601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.191935 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.192145 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.192150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.193440 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.193441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.194619 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.194622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.195186 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.195187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.195341 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.196312 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.196318 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.199747 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.202419 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.202421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.202810 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.203053 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.203054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.203055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.203117 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.203453 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.203454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.203501 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.203923 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.204042 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":9,"thread_start_us":95,"threads_started":1}
I20260620 08:04:35.204636 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.189764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:33301 (local address 127.31.115.252:44943)
0620 08:04:35.189817 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.189820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.190161 (+ 341us) server_negotiation.cc:407] Connection header received
0620 08:04:35.191386 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.191387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.191415 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.191454 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.192262 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.193037 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.194727 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.195043 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.195087 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.200033 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.200035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.200035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.200044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.202261 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202907 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.202909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.202910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.202986 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.203258 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.203259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.203260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.203422 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.203502 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.204403 (+ 901us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.204526 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:04:35.204300 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.189914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:44889)
0620 08:04:35.190075 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.190086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.191182 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.191503 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.191509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.191965 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.192146 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.192153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.193174 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.193175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.194616 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.194619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.195183 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.195184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.195341 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.196082 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.196091 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.199196 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.201660 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.201671 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.201673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.202067 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.202360 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.202361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.202541 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.202947 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.202956 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.203167 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.203731 (+ 564us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.203974 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":108,"thread_start_us":102,"threads_started":1}
I20260620 08:04:35.204906 6965 heartbeater.cc:347] Connected to a master server at 127.31.115.253:35265
I20260620 08:04:35.205025 6965 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.205307 6964 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34223
I20260620 08:04:35.205353 6964 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.205672 6965 heartbeater.cc:511] Master 127.31.115.253:35265 requested a full tablet report, sending...
I20260620 08:04:35.205672 6964 heartbeater.cc:511] Master 127.31.115.254:34223 requested a full tablet report, sending...
I20260620 08:04:35.206274 6455 ts_manager.cc:194] Registered new tserver with Master: fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645)
I20260620 08:04:35.206291 5853 ts_manager.cc:194] Re-registered known tserver with Master: fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645)
I20260620 08:04:35.206835 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.189703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.31.115.194:33301)
0620 08:04:35.189992 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.190038 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.191180 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.191553 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.191554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.191980 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.192146 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.192152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.193266 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.193267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.194616 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.194619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.195184 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.195184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.195341 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.196166 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.196170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.199924 (+ 3754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.202363 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.202364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.202364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.202799 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.203075 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.203076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.203076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.203145 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.203544 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.203545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.203592 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.204339 (+ 747us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.206762 (+ 2423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":128,"thread_start_us":92,"threads_started":1}
I20260620 08:04:35.206863 5853 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:44889
I20260620 08:04:35.207271 6963 heartbeater.cc:347] Connected to a master server at 127.31.115.252:44943
I20260620 08:04:35.207327 6963 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.207410 6963 heartbeater.cc:511] Master 127.31.115.252:44943 requested a full tablet report, sending...
I20260620 08:04:35.207737 6645 ts_manager.cc:194] Registered new tserver with Master: fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645)
W20260620 08:04:35.288537 6970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:35.288760 6970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:35.288787 6970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:35.289974 6970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:35.290015 6970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:35.290042 6970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:35.291491 6970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:41283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=45545
--tserver_master_addrs=127.31.115.254:34223,127.31.115.253:35265,127.31.115.252:44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:35.291709 6970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:35.291890 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:35.297320 6975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:35.297362 6976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:35.297320 6978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:35.297534 6970 server_base.cc:1034] running on GCE node
I20260620 08:04:35.297772 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:35.297999 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:35.299218 6970 hybrid_clock.cc:648] HybridClock initialized: now 1781942675299197 us; error 26 us; skew 500 ppm
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:35.301906 6970 init.cc:377] Logged in from keytab as oryx/127.31.115.195@KRBTEST.COM (short username oryx)
I20260620 08:04:35.302389 6970 webserver.cc:460] Webserver started at http://127.31.115.195:45545/ using document root <none> and password file <none>
I20260620 08:04:35.302649 6970 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:35.302706 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:35.304179 6970 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:35.304888 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.305088 6970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:35.305179 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "71c8a779dc0b4df9af438cf8e2ca54a3"
format_stamp: "Formatted at 2026-06-20 08:04:34 on dist-test-slave-c4w8"
server_key: "6ab4b30ca71f7be0ef9af88341c4bf56"
server_key_iv: "0bba6767aba65eea72a125b7ac0fbb58"
server_key_version: "encryptionkey@0"
I20260620 08:04:35.305518 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:35.314594 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:35.317121 6970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:35.317241 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:35.317459 6970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:35.317812 6970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:35.317848 6970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.317871 6970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:35.317883 6970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:35.323685 6970 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41283
I20260620 08:04:35.323740 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41283 every 8 connection(s)
I20260620 08:04:35.324136 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:35.326980 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 6970
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.252@KRBTEST.COM
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.253@KRBTEST.COM
Jun 20 08:04:35 dist-test-slave-c4w8 krb5kdc[5807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:35.339061 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.328536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.31.115.254:34223)
0620 08:04:35.328693 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.328696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.328705 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:35.328729 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.328730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.328743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.328813 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.329633 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.330434 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.331989 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.332294 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.332387 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.333972 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.333973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.333973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.333980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.336774 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.337903 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.337905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.337905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.337965 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.338378 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.338380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.338380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.338545 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.338647 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.338791 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.338880 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260620 08:04:35.340318 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.328395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.31.115.252:44943)
0620 08:04:35.328520 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.328522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.328681 (+ 159us) server_negotiation.cc:407] Connection header received
0620 08:04:35.328781 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.328781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.328793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.328830 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.330260 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.330996 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.333205 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.333543 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.333591 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.335144 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.335145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.335145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.335151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.337084 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.337579 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.337581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.337581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.337678 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.338087 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.338088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.338088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.338254 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.338341 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.340112 (+ 1771us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.340205 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260620 08:04:35.341874 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.324992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:33321 (local address 127.31.115.253:35265)
0620 08:04:35.325045 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.325048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.325500 (+ 452us) server_negotiation.cc:407] Connection header received
0620 08:04:35.326484 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.326485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.326498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.326567 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.327826 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.328663 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.331053 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.331340 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.331404 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.337402 (+ 5998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.337404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.337404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.337413 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.339978 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340536 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.340537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.340537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.340594 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340906 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.340907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.340907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.341044 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.341134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.341719 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.341794 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:35.341890 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.324980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.31.115.195:33321)
0620 08:04:35.325442 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.325452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.326380 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.326670 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.326675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.327210 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.327732 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.327735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.328775 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.328776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.330968 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.330971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.331485 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.331486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.331654 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.334983 (+ 3329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.334988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.337204 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.340079 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.340081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.340457 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.340664 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.340665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.340830 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.341136 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.341144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.341328 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.341430 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.341654 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":1717,"thread_start_us":102,"threads_started":1}
I20260620 08:04:35.342116 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.324980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.195:46379)
0620 08:04:35.325200 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.325245 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.326370 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.326809 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.326810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.327210 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.327537 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.327544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.332072 (+ 4528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.332073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.333315 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.333319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.334200 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.334201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.334281 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.334633 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.334638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.337401 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.340125 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.340126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.340613 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.341080 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.341081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.341081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.341135 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.341816 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.341817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.341854 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.341912 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.342009 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":16,"thread_start_us":120,"threads_started":1}
I20260620 08:04:35.342463 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.325317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.31.115.195:50237)
0620 08:04:35.326580 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.326589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.326638 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.326755 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.326755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.327335 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:35.327537 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.327544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.328706 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.328707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.330141 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.330145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.332339 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.332340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.332406 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.332860 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:35.332871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:35.335026 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:35.339966 (+ 4940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.339978 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.339982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.340446 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.340828 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340828 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:35.340829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:35.340893 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:35.341411 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:35.341412 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:35.341451 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.341502 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.342365 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":756,"thread_start_us":60,"threads_started":1}
I20260620 08:04:35.342653 7101 heartbeater.cc:347] Connected to a master server at 127.31.115.253:35265
I20260620 08:04:35.342748 7101 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.342797 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.325409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:50237 (local address 127.31.115.252:44943)
0620 08:04:35.325466 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.325468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.326638 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 08:04:35.326685 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.326686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.326697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.326749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.327677 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.328429 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.331925 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.332221 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.332279 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.335130 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.335131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.335132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.335139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.339134 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340698 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.340699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.340699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.340758 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.341209 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.341210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.341210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.341336 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.341387 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.342635 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.342715 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:04:35.342937 7101 heartbeater.cc:511] Master 127.31.115.253:35265 requested a full tablet report, sending...
I20260620 08:04:35.343060 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.325016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:46379 (local address 127.31.115.254:34223)
0620 08:04:35.325227 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.325229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.325345 (+ 116us) server_negotiation.cc:407] Connection header received
0620 08:04:35.326485 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.326486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.326498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.326629 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.331076 (+ 4447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.331819 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.333470 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.333832 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.333896 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.337491 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.337492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.337493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.337500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.339871 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.340955 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.340956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.340956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.341010 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.341582 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.341583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.341584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.341728 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.341780 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.342908 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.342982 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260620 08:04:35.343087 7099 heartbeater.cc:347] Connected to a master server at 127.31.115.252:44943
I20260620 08:04:35.343127 7099 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.343199 7099 heartbeater.cc:511] Master 127.31.115.252:44943 requested a full tablet report, sending...
I20260620 08:04:35.343324 7100 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34223
I20260620 08:04:35.343374 7100 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:35.341138 5853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:04:35.343431 7100 heartbeater.cc:511] Master 127.31.115.254:34223 requested a full tablet report, sending...
I20260620 08:04:35.343533 6645 ts_manager.cc:194] Registered new tserver with Master: 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283)
W20260620 08:04:35.343623 5853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:35.343668 5852 ts_manager.cc:194] Re-registered known tserver with Master: 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283)
I20260620 08:04:35.343886 6455 ts_manager.cc:194] Registered new tserver with Master: 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283)
I20260620 08:04:35.344098 5852 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:46379
I20260620 08:04:35.345741 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.329259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.31.115.253:35265)
0620 08:04:35.329375 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.329377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.329385 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:35.329410 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.329410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.329419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.329496 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.330183 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.330787 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.331804 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.332570 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.332634 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.338174 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.338175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.338176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.338183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.341245 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.344673 (+ 3428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.344674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.344674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.344757 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.345120 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.345121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.345121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.345253 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.345348 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.345525 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.345651 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260620 08:04:35.352466 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.348185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41283 (local address 127.0.0.1:43052)
0620 08:04:35.348241 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.348253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.348323 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.348778 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.348780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.348781 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:35.348828 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.348830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.349870 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.349871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.350803 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.350806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.352211 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.352211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.352274 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.352278 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.352344 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.352389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:04:35.352672 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.348297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.31.115.195:41283)
0620 08:04:35.348623 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.348629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.348641 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:04:35.348680 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.348682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.348698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.348781 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:35.348882 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.349791 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.351525 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.352127 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.352230 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.352290 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.352358 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.352392 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":95,"threads_started":1}
I20260620 08:04:35.352671 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.348306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:44645 (local address 127.0.0.1:53708)
0620 08:04:35.348435 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.348443 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.348481 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.349010 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.349010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.349012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:35.349048 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.349050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.350552 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.350553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.351444 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.351447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.352427 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.352428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.352489 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.352492 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.352549 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.352595 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260620 08:04:35.352905 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.348570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.31.115.194:44645)
0620 08:04:35.348839 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.348846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.348856 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:35.348897 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.348899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.348914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.349003 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:35.349133 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.350412 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.351517 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.352241 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.352340 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.352391 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.352777 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.352823 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260620 08:04:35.353381 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5d10be6e46452283e6cc009147fc5b (DEFAULT_TABLE table=test-table [id=4ffa9aac39b94f46b55e8fa20281c538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:35.353507 6897 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5d10be6e46452283e6cc009147fc5b (DEFAULT_TABLE table=test-table [id=4ffa9aac39b94f46b55e8fa20281c538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:35.353796 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5d10be6e46452283e6cc009147fc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:35.353863 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.348539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40327 (local address 127.0.0.1:55528)
0620 08:04:35.348661 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.348671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.348721 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.349927 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.349928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.349931 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:35.349977 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.349979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.351292 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.351293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.352399 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.352402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.353648 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.353649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.353708 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.353712 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.353751 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.353790 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 08:04:35.353888 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.348809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.31.115.193:40327)
0620 08:04:35.349750 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.349754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.349765 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:35.349803 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.349806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.349821 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.350045 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:35.350103 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.351200 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.352488 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.353533 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.353677 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.353736 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.353773 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.353811 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":81,"threads_started":1}
I20260620 08:04:35.353926 6897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5d10be6e46452283e6cc009147fc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:35.354488 6759 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5d10be6e46452283e6cc009147fc5b (DEFAULT_TABLE table=test-table [id=4ffa9aac39b94f46b55e8fa20281c538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:35.354810 6759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5d10be6e46452283e6cc009147fc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:35.357446 7125 tablet_bootstrap.cc:492] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Bootstrap starting.
I20260620 08:04:35.358121 7127 tablet_bootstrap.cc:492] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Bootstrap starting.
I20260620 08:04:35.358608 7125 tablet_bootstrap.cc:654] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:35.358788 7127 tablet_bootstrap.cc:654] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:35.358968 7125 log.cc:824] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:35.359134 7127 log.cc:824] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:35.359344 7126 tablet_bootstrap.cc:492] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Bootstrap starting.
I20260620 08:04:35.359947 7126 tablet_bootstrap.cc:654] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:35.359965 7127 tablet_bootstrap.cc:492] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: No bootstrap required, opened a new log
I20260620 08:04:35.360050 7127 ts_tablet_manager.cc:1397] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:35.360234 7126 log.cc:824] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:35.360785 7125 tablet_bootstrap.cc:492] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: No bootstrap required, opened a new log
I20260620 08:04:35.360860 7125 ts_tablet_manager.cc:1397] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:04:35.360895 7126 tablet_bootstrap.cc:492] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: No bootstrap required, opened a new log
I20260620 08:04:35.360961 7126 ts_tablet_manager.cc:1397] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:35.362134 7127 raft_consensus.cc:348] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.362293 7127 raft_consensus.cc:374] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:35.362334 7127 raft_consensus.cc:729] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afebc8d250f49698ee5741ed84d845c, State: Initialized, Role: FOLLOWER
I20260620 08:04:35.362463 7127 consensus_queue.cc:260] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.362710 7127 ts_tablet_manager.cc:1428] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:35.362847 6826 heartbeater.cc:503] Master 127.31.115.254:34223 was elected leader, sending a full tablet report...
I20260620 08:04:35.362897 7125 raft_consensus.cc:348] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.362914 7126 raft_consensus.cc:348] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.363063 7125 raft_consensus.cc:374] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:35.363076 7126 raft_consensus.cc:374] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:35.363111 7126 raft_consensus.cc:729] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4e61e11f49480db968cbb18a9b438f, State: Initialized, Role: FOLLOWER
I20260620 08:04:35.363104 7125 raft_consensus.cc:729] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c8a779dc0b4df9af438cf8e2ca54a3, State: Initialized, Role: FOLLOWER
I20260620 08:04:35.363247 7125 consensus_queue.cc:260] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.363260 7126 consensus_queue.cc:260] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.363521 7126 ts_tablet_manager.cc:1428] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:35.363521 7125 ts_tablet_manager.cc:1428] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:35.363636 7100 heartbeater.cc:503] Master 127.31.115.254:34223 was elected leader, sending a full tablet report...
I20260620 08:04:35.363896 6964 heartbeater.cc:503] Master 127.31.115.254:34223 was elected leader, sending a full tablet report...
I20260620 08:04:35.392410 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.389360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43204)
0620 08:04:35.389427 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.389440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.389504 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.389756 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.389757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.389758 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:35.389810 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.389813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.390481 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.390482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.391288 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.391292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.392024 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.392026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.392129 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.392134 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.392236 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.392301 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:04:35.392424 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.389448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.31.115.252:44943)
0620 08:04:35.389547 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.389549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.389558 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:35.389583 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.389584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.389663 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.389740 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:35.389879 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.390388 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.391397 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.391898 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.391993 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.392063 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.392247 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.392338 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 08:04:35.392930 6655 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 08:04:35.394291 6684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faec5a8699d8449caee9416190eaabe5. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:35.394397 6684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: This master's current role is: LEARNER
W20260620 08:04:35.440009 6966 tablet.cc:2369] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:35.443986 7132 raft_consensus.cc:482] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:35.444130 7132 raft_consensus.cc:504] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.444446 7132 leader_election.cc:288] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645), 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:35.447917 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.444678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40327 (local address 127.31.115.195:34201)
0620 08:04:35.444853 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.444864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.444929 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.445064 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.445064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.445066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:35.445124 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.445125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.445963 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.445964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.446831 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.446834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.447473 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.447474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.447722 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.447729 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.447794 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.447852 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":102}
I20260620 08:04:35.447980 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.444749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:34201 (local address 127.31.115.193:40327)
0620 08:04:35.444805 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.444807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.444945 (+ 138us) server_negotiation.cc:407] Connection header received
0620 08:04:35.444976 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.444977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.444987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.445046 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:35.445202 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.445881 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.446936 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.447374 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.447433 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.447440 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.447833 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.447887 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:04:35.448094 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.444678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:44645 (local address 127.31.115.195:40609)
0620 08:04:35.444744 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:35.444757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:35.444801 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:35.444956 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:35.444957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:35.444960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:35.445387 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.445390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.446266 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.446267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.447024 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:35.447027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.447822 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.447824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:35.447917 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.447921 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:35.447982 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:04:35.448028 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 08:04:35.448355 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5d10be6e46452283e6cc009147fc5b" candidate_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4afebc8d250f49698ee5741ed84d845c" is_pre_election: true
I20260620 08:04:35.448558 6779 raft_consensus.cc:2454] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c8a779dc0b4df9af438cf8e2ca54a3 in term 0.
I20260620 08:04:35.448611 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.444754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:40609 (local address 127.31.115.194:44645)
0620 08:04:35.444813 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.444815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.444824 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:35.444856 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.444857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.444869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.444938 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:35.445503 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.446165 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.447132 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.447582 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.447777 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.447792 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.448488 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.448539 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:04:35.448889 6987 leader_election.cc:302] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afebc8d250f49698ee5741ed84d845c, 71c8a779dc0b4df9af438cf8e2ca54a3; no voters:
I20260620 08:04:35.448889 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5d10be6e46452283e6cc009147fc5b" candidate_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4e61e11f49480db968cbb18a9b438f" is_pre_election: true
I20260620 08:04:35.449046 6917 raft_consensus.cc:2454] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c8a779dc0b4df9af438cf8e2ca54a3 in term 0.
I20260620 08:04:35.449038 7132 raft_consensus.cc:2781] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:35.449157 7132 raft_consensus.cc:482] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:35.449196 7132 raft_consensus.cc:3037] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:35.450001 7132 raft_consensus.cc:504] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.450193 7132 leader_election.cc:288] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [CANDIDATE]: Term 1 election: Requested vote from peers fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645), 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:35.450349 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5d10be6e46452283e6cc009147fc5b" candidate_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4e61e11f49480db968cbb18a9b438f"
I20260620 08:04:35.450392 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5d10be6e46452283e6cc009147fc5b" candidate_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4afebc8d250f49698ee5741ed84d845c"
I20260620 08:04:35.450446 6917 raft_consensus.cc:3037] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:35.450510 6779 raft_consensus.cc:3037] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:35.451290 6917 raft_consensus.cc:2454] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c8a779dc0b4df9af438cf8e2ca54a3 in term 1.
I20260620 08:04:35.451301 6779 raft_consensus.cc:2454] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c8a779dc0b4df9af438cf8e2ca54a3 in term 1.
I20260620 08:04:35.451514 6987 leader_election.cc:302] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afebc8d250f49698ee5741ed84d845c, 71c8a779dc0b4df9af438cf8e2ca54a3; no voters:
I20260620 08:04:35.451638 7132 raft_consensus.cc:2781] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:35.451810 7132 raft_consensus.cc:686] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 LEADER]: Becoming Leader. State: Replica: 71c8a779dc0b4df9af438cf8e2ca54a3, State: Running, Role: LEADER
I20260620 08:04:35.451964 7132 consensus_queue.cc:237] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } }
I20260620 08:04:35.452844 5853 catalog_manager.cc:5495] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195). New cstate: current_term: 1 leader_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:35.453490 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: NON_VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: true } } }
I20260620 08:04:35.453598 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:35.454846 6605 raft_consensus.cc:1050] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: attempting to promote NON_VOTER 9cb7362dc7c4489aac838c9083262776 to VOTER
I20260620 08:04:35.454988 6605 consensus_queue.cc:237] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } }
I20260620 08:04:35.455384 6465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Refusing update from remote peer faec5a8699d8449caee9416190eaabe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 08:04:35.455415 6655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 LEARNER]: Refusing update from remote peer faec5a8699d8449caee9416190eaabe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 08:04:35.455580 7139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 08:04:35.455757 6605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 08:04:35.456303 7139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9cb7362dc7c4489aac838c9083262776 (127.31.115.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.456485 6465 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9cb7362dc7c4489aac838c9083262776 (127.31.115.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.457206 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.457316 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:35.457296 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.457378 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.457387 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:35.457448 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:35.457408 6655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9cb7362dc7c4489aac838c9083262776 (127.31.115.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.457679 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.457799 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:04:35.458344 6684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "faec5a8699d8449caee9416190eaabe5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:35.458434 6684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:04:35.470410 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.467876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.31.115.195:41283)
0620 08:04:35.467944 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.467950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.467957 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:35.467991 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.467992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.468010 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.468094 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:35.468224 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.468704 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.469477 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.469780 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.469859 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.470037 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:35.470163 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:35.470263 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.470281 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.470304 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:04:35.472451 6917 raft_consensus.cc:1261] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Refusing update from remote peer 71c8a779dc0b4df9af438cf8e2ca54a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:35.472507 6779 raft_consensus.cc:1261] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Refusing update from remote peer 71c8a779dc0b4df9af438cf8e2ca54a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:35.472872 7132 consensus_queue.cc:1035] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:35.473037 7137 consensus_queue.cc:1035] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:35.478571 7143 mvcc.cc:205] Tried to move back new op lower bound from 7298837198732275712 to 7298837198652002304. Current Snapshot: MvccSnapshot[applied={T|T < 7298837198732275712 or (T in {7298837198732275712})}]
I20260620 08:04:35.479534 5852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:35.479647 5852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:35.481010 5852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faec5a8699d8449caee9416190eaabe5: Sending DeleteTablet for 3 replicas of tablet 4a5d10be6e46452283e6cc009147fc5b
I20260620 08:04:35.481587 6759 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5d10be6e46452283e6cc009147fc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:35 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:55528
I20260620 08:04:35.481556 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5d10be6e46452283e6cc009147fc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:35 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:43052
I20260620 08:04:35.481853 7153 tablet_replica.cc:330] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: stopping tablet replica
I20260620 08:04:35.481935 6897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5d10be6e46452283e6cc009147fc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:35 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53708
I20260620 08:04:35.481971 7153 raft_consensus.cc:2229] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:35.482086 7153 raft_consensus.cc:2258] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:35.482250 7152 tablet_replica.cc:330] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: stopping tablet replica
I20260620 08:04:35.482378 7152 raft_consensus.cc:2229] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:35.482474 7152 raft_consensus.cc:2258] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:35.482805 7155 tablet_replica.cc:330] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: stopping tablet replica
I20260620 08:04:35.482916 7155 raft_consensus.cc:2229] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:35.482975 7152 ts_tablet_manager.cc:1905] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:35.483021 7155 raft_consensus.cc:2258] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:35.483516 7155 ts_tablet_manager.cc:1905] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:35.483925 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 5821
I20260620 08:04:35.485183 7152 ts_tablet_manager.cc:1918] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:35.485246 7152 log.cc:1196] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/4a5d10be6e46452283e6cc009147fc5b
I20260620 08:04:35.485536 7152 ts_tablet_manager.cc:1939] T 4a5d10be6e46452283e6cc009147fc5b P 71c8a779dc0b4df9af438cf8e2ca54a3: Deleting consensus metadata
I20260620 08:04:35.485884 7155 ts_tablet_manager.cc:1918] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:35.485966 7155 log.cc:1196] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/4a5d10be6e46452283e6cc009147fc5b
I20260620 08:04:35.486238 7155 ts_tablet_manager.cc:1939] T 4a5d10be6e46452283e6cc009147fc5b P fe4e61e11f49480db968cbb18a9b438f: Deleting consensus metadata
W20260620 08:04:35.490727 6853 connection.cc:535] server connection from 127.0.0.1:53708 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 08:04:35.490947 6986 connection.cc:535] server connection from 127.0.0.1:43052 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:04:35.491118 7153 ts_tablet_manager.cc:1905] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:35.493043 7153 ts_tablet_manager.cc:1918] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:35.493204 7153 log.cc:1196] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/4a5d10be6e46452283e6cc009147fc5b
I20260620 08:04:35.493561 7153 ts_tablet_manager.cc:1939] T 4a5d10be6e46452283e6cc009147fc5b P 4afebc8d250f49698ee5741ed84d845c: Deleting consensus metadata
W20260620 08:04:35.493997 6714 connection.cc:441] server connection from 127.0.0.1:55528 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55528 (request call id 1) could send its response
I20260620 08:04:35.498736 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.482349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.31.115.252:44943)
0620 08:04:35.482393 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.482395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.482745 (+ 350us) server_negotiation.cc:407] Connection header received
0620 08:04:35.482771 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.482771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.482784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.482834 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.490819 (+ 7985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.491564 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.492841 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.493173 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.493218 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.495017 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.495018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.495019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.495026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.496709 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.497505 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.497507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.497507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.497568 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.497814 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.497815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.497815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.498237 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.498317 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.498543 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.498621 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:04:35.501252 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.492697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.31.115.252:44943)
0620 08:04:35.492742 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.492744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.493186 (+ 442us) server_negotiation.cc:407] Connection header received
0620 08:04:35.493284 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.493285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.493301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.493333 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.494221 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.495006 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.496174 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.496491 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.496538 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.498306 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.498307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.498307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.498313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.499985 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.500403 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.500404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.500405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.500460 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.500693 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.500694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.500694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.500809 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.500917 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.501024 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.501103 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:35.502422 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:35.494341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.31.115.253:35265)
0620 08:04:35.494387 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:35.494389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:35.494682 (+ 293us) server_negotiation.cc:407] Connection header received
0620 08:04:35.494780 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:35.494781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:35.494794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:35.494825 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:35.495476 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.496244 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.497297 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:35.497641 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:35.497682 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:35.499380 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:35.499382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:35.499382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:35.499391 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:35.501097 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.501728 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.501729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.501729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.501783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:35.501983 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:35.501984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:35.501984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:35.502086 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:35.502165 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:35.502292 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 08:04:35.502360 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:36.477725 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.477391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:59405)
0620 08:04:36.477548 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.477663 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260620 08:04:36.477986 6964 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:34223 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.478425 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.478160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.195:33465)
0620 08:04:36.478282 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.478356 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
W20260620 08:04:36.478585 7100 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:34223 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.479257 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.479014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:48323)
0620 08:04:36.479140 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.479211 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
W20260620 08:04:36.479490 6826 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:34223 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.982332 7166 raft_consensus.cc:482] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faec5a8699d8449caee9416190eaabe5)
I20260620 08:04:36.982577 7166 raft_consensus.cc:504] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } }
I20260620 08:04:36.982594 7167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faec5a8699d8449caee9416190eaabe5)
I20260620 08:04:36.982718 7167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } }
I20260620 08:04:36.982878 7166 leader_election.cc:288] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223), 9cb7362dc7c4489aac838c9083262776 (127.31.115.252:44943)
I20260620 08:04:36.983325 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.982961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36858)
0620 08:04:36.983185 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.983266 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":64,"threads_started":1}
I20260620 08:04:36.983345 7167 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223), e869551e011b4ea59efb71ecc8d0169e (127.31.115.253:35265)
W20260620 08:04:36.983485 6441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.983943 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.983113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36860)
0620 08:04:36.983359 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.983900 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":56,"threads_started":1}
W20260620 08:04:36.984058 6631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.984476 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.984360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36882)
0620 08:04:36.984411 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.984447 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:04:36.984624 6631 leader_election.cc:334] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223): Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.984880 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.983892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36872)
0620 08:04:36.984823 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.984853 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912}
W20260620 08:04:36.984964 6441 leader_election.cc:334] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223): Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.986776 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.983201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.31.115.252:44943)
0620 08:04:36.983441 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:36.983443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:36.983451 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:36.983482 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:36.983484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:36.983496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:36.983642 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:36.983829 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.984715 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.985951 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.986381 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.986483 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:36.986494 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:36.986659 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 08:04:36.986708 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":48,"threads_started":1}
I20260620 08:04:36.986783 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.983055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:44943 (local address 127.0.0.1:43242)
0620 08:04:36.983193 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.983205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:36.983277 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:36.983669 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:36.983670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:36.983673 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:36.983736 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:36.983738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.984794 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.984796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:36.985856 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:36.985859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.986483 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.986485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:36.986557 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:36.986561 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:36.986651 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:04:36.986703 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260620 08:04:36.986879 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.983167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:35265 (local address 127.0.0.1:40108)
0620 08:04:36.983447 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.983459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:36.983522 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:36.983935 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:36.983936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:36.983938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:36.983981 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:36.983983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.984841 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.984843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:36.985987 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:36.985990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.986570 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.986571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:36.986633 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:36.986636 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:36.986711 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:04:36.986761 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":30,"threads_started":1}
I20260620 08:04:36.987110 6655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e869551e011b4ea59efb71ecc8d0169e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9cb7362dc7c4489aac838c9083262776" is_pre_election: true
I20260620 08:04:36.987232 6655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e869551e011b4ea59efb71ecc8d0169e in term 1.
I20260620 08:04:36.987480 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.983290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.31.115.253:35265)
0620 08:04:36.983385 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:36.983388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:36.983532 (+ 144us) server_negotiation.cc:407] Connection header received
0620 08:04:36.983824 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:36.983825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:36.983836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:36.983893 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:36.984059 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.984774 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.986078 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:36.986467 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:36.986566 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:36.986575 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:36.986718 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 08:04:36.986761 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":34,"threads_started":1}
I20260620 08:04:36.987524 6442 leader_election.cc:302] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cb7362dc7c4489aac838c9083262776, e869551e011b4ea59efb71ecc8d0169e; no voters: faec5a8699d8449caee9416190eaabe5
I20260620 08:04:36.987659 7166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 08:04:36.987704 7166 raft_consensus.cc:482] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Starting leader election (detected failure of leader faec5a8699d8449caee9416190eaabe5)
I20260620 08:04:36.987726 7166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:04:36.987753 6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cb7362dc7c4489aac838c9083262776" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e869551e011b4ea59efb71ecc8d0169e" is_pre_election: true
I20260620 08:04:36.988477 7166 raft_consensus.cc:504] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } }
I20260620 08:04:36.988629 7166 leader_election.cc:288] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [CANDIDATE]: Term 2 election: Requested vote from peers faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223), 9cb7362dc7c4489aac838c9083262776 (127.31.115.252:44943)
I20260620 08:04:36.988638 6465 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9cb7362dc7c4489aac838c9083262776 in current term 2: Already voted for candidate e869551e011b4ea59efb71ecc8d0169e in this term.
I20260620 08:04:36.988811 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.988717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36884)
0620 08:04:36.988759 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.988784 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 08:04:36.988838 6655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e869551e011b4ea59efb71ecc8d0169e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9cb7362dc7c4489aac838c9083262776"
I20260620 08:04:36.988900 6629 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cb7362dc7c4489aac838c9083262776; no voters: e869551e011b4ea59efb71ecc8d0169e, faec5a8699d8449caee9416190eaabe5
I20260620 08:04:36.988919 6655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:04:36.989295 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:36.989175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36890)
0620 08:04:36.989239 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:36.989272 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 08:04:36.989378 6441 leader_election.cc:334] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223): Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
I20260620 08:04:36.989760 6655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e869551e011b4ea59efb71ecc8d0169e in term 2.
I20260620 08:04:36.989836 7167 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:04:36.989918 6442 leader_election.cc:302] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cb7362dc7c4489aac838c9083262776, e869551e011b4ea59efb71ecc8d0169e; no voters: faec5a8699d8449caee9416190eaabe5
I20260620 08:04:36.990025 7166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:04:36.990154 7166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [term 2 LEADER]: Becoming Leader. State: Replica: e869551e011b4ea59efb71ecc8d0169e, State: Running, Role: LEADER
I20260620 08:04:36.990257 7166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } }
I20260620 08:04:36.990624 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e869551e011b4ea59efb71ecc8d0169e. Latest consensus state: current_term: 2 leader_uuid: "e869551e011b4ea59efb71ecc8d0169e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:36.990720 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:36.990875 7178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:36.991330 7178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:36.991393 7178 catalog_manager.cc:1259] Loaded cluster ID: 55dc5a175cee4c7b9141fd115dece84c
I20260620 08:04:36.991410 7178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:36.991533 7178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:36.991585 7178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Loaded TSK: 0
I20260620 08:04:36.991729 7178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:37.040105 6455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:37.041783 6455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:37.043094 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.042904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36906)
0620 08:04:37.043011 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.043054 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 08:04:37.043069 6655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [term 2 FOLLOWER]: Refusing update from remote peer e869551e011b4ea59efb71ecc8d0169e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 08:04:37.043591 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.043473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.0.0.1:36922)
0620 08:04:37.043525 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.043555 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:04:37.043581 7166 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 08:04:37.043737 6441 consensus_peers.cc:487] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e -> Peer faec5a8699d8449caee9416190eaabe5 (127.31.115.254:34223): Couldn't send request to peer faec5a8699d8449caee9416190eaabe5. Status: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 08:04:37.044373 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e869551e011b4ea59efb71ecc8d0169e. Latest consensus state: current_term: 2 leader_uuid: "e869551e011b4ea59efb71ecc8d0169e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:37.044504 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:04:37.045089 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e869551e011b4ea59efb71ecc8d0169e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:37.045192 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:37.045428 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e869551e011b4ea59efb71ecc8d0169e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "faec5a8699d8449caee9416190eaabe5" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34223 } } peers { permanent_uuid: "e869551e011b4ea59efb71ecc8d0169e" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 35265 } attrs { promote: false } } peers { permanent_uuid: "9cb7362dc7c4489aac838c9083262776" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 44943 } attrs { promote: false } } }
I20260620 08:04:37.045544 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb7362dc7c4489aac838c9083262776 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:04:37.050483 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.047068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40327 (local address 127.0.0.1:55540)
0620 08:04:37.047240 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.047250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:37.047304 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:37.047619 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:37.047620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:37.047622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:37.047673 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.047674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.048534 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.048535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.049421 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.049423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.050205 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.050206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.050271 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.050275 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:37.050335 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:04:37.050378 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":30}
I20260620 08:04:37.050487 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.047385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.31.115.193:40327)
0620 08:04:37.047497 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:37.047499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:37.047507 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:37.047534 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:37.047534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:37.047545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:37.047615 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:37.047764 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.048470 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.049514 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.050137 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.050213 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.050223 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:37.050348 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 08:04:37.050393 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260620 08:04:37.050729 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.047019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41283 (local address 127.0.0.1:43076)
0620 08:04:37.047226 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.047238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:37.047287 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:37.047487 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:37.047487 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:37.047489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:37.047542 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.047543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.048472 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.048474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.049731 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.049733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.050471 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.050472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.050536 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.050542 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:04:37.050595 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:04:37.050642 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":75}
I20260620 08:04:37.050755 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.047112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:44645 (local address 127.0.0.1:53720)
0620 08:04:37.047363 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.047373 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:37.047418 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:37.047826 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:37.047827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:37.047828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:37.047861 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.047862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.048509 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.048510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.049731 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.049733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.050502 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.050503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.050560 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.050565 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:37.050628 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:04:37.050672 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":26,"thread_start_us":78,"threads_started":1}
I20260620 08:04:37.050765 6759 tablet_service.cc:1467] Processing CreateTablet for tablet 74ce7d1a8b7f45c6a99825e9735a04b5 (DEFAULT_TABLE table=test-table [id=7e372963c61041599cec6b93539a570a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:37.050904 6759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ce7d1a8b7f45c6a99825e9735a04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:37.050949 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.047179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.31.115.194:44645)
0620 08:04:37.047698 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:37.047700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:37.047709 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:37.047748 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:37.047749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:37.047759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:37.047816 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:37.047909 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.048403 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.049837 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.050401 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.050462 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.050469 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:37.050832 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 08:04:37.050879 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":123,"threads_started":1}
I20260620 08:04:37.051229 6897 tablet_service.cc:1467] Processing CreateTablet for tablet 74ce7d1a8b7f45c6a99825e9735a04b5 (DEFAULT_TABLE table=test-table [id=7e372963c61041599cec6b93539a570a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:37.051395 6897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ce7d1a8b7f45c6a99825e9735a04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:37.052340 7189 tablet_bootstrap.cc:492] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: Bootstrap starting.
I20260620 08:04:37.052728 7189 tablet_bootstrap.cc:654] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:37.053097 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.047085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.31.115.195:41283)
0620 08:04:37.047204 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:37.047206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:37.047348 (+ 142us) server_negotiation.cc:407] Connection header received
0620 08:04:37.047383 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:37.047385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:37.047399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:37.047486 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:37.047593 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.048378 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.049820 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.050378 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.052948 (+ 2570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.052963 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:37.052986 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 08:04:37.053039 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260620 08:04:37.053262 7190 tablet_bootstrap.cc:492] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Bootstrap starting.
I20260620 08:04:37.053318 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 74ce7d1a8b7f45c6a99825e9735a04b5 (DEFAULT_TABLE table=test-table [id=7e372963c61041599cec6b93539a570a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:37.053434 7189 tablet_bootstrap.cc:492] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: No bootstrap required, opened a new log
I20260620 08:04:37.053442 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ce7d1a8b7f45c6a99825e9735a04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:37.053489 7189 ts_tablet_manager.cc:1397] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:37.053599 7190 tablet_bootstrap.cc:654] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:37.053663 7189 raft_consensus.cc:348] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.053768 7189 raft_consensus.cc:374] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:37.053791 7189 raft_consensus.cc:729] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afebc8d250f49698ee5741ed84d845c, State: Initialized, Role: FOLLOWER
I20260620 08:04:37.053862 7189 consensus_queue.cc:260] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.054020 7189 ts_tablet_manager.cc:1428] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:37.054948 6827 heartbeater.cc:503] Master 127.31.115.253:35265 was elected leader, sending a full tablet report...
I20260620 08:04:37.055043 7190 tablet_bootstrap.cc:492] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: No bootstrap required, opened a new log
I20260620 08:04:37.055094 7190 ts_tablet_manager.cc:1397] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:37.055119 7195 tablet_bootstrap.cc:492] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Bootstrap starting.
I20260620 08:04:37.055267 7190 raft_consensus.cc:348] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.055389 7190 raft_consensus.cc:374] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:37.055419 7190 raft_consensus.cc:729] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4e61e11f49480db968cbb18a9b438f, State: Initialized, Role: FOLLOWER
I20260620 08:04:37.055475 7190 consensus_queue.cc:260] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.055645 7190 ts_tablet_manager.cc:1428] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:37.055750 7195 tablet_bootstrap.cc:654] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:37.055845 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.054329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:40119)
0620 08:04:37.055747 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.055795 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":82,"threads_started":1}
I20260620 08:04:37.056285 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.056070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:60291)
0620 08:04:37.056216 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.056243 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260620 08:04:37.056280 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.056041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:53129)
0620 08:04:37.056196 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.056236 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260620 08:04:37.056583 6965 heartbeater.cc:503] Master 127.31.115.253:35265 was elected leader, sending a full tablet report...
I20260620 08:04:37.056802 7195 tablet_bootstrap.cc:492] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: No bootstrap required, opened a new log
I20260620 08:04:37.056851 7195 ts_tablet_manager.cc:1397] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:37.056965 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.056669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:34619)
0620 08:04:37.056768 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.056795 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 08:04:37.056965 7195 raft_consensus.cc:348] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.057039 7195 raft_consensus.cc:374] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:37.057061 7195 raft_consensus.cc:729] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c8a779dc0b4df9af438cf8e2ca54a3, State: Initialized, Role: FOLLOWER
I20260620 08:04:37.057118 7195 consensus_queue.cc:260] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.057255 7195 ts_tablet_manager.cc:1428] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:37.057725 7101 heartbeater.cc:503] Master 127.31.115.253:35265 was elected leader, sending a full tablet report...
I20260620 08:04:37.057845 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.057434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.195:54287)
0620 08:04:37.057557 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.057697 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":79,"threads_started":1}
W20260620 08:04:37.076817 7102 tablet.cc:2369] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:37.124513 7196 raft_consensus.cc:482] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:37.124624 7196 raft_consensus.cc:504] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.125058 7196 leader_election.cc:288] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283), 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:37.129259 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.125406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:40957 (local address 127.31.115.193:40327)
0620 08:04:37.125455 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:37.125457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:37.125467 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:37.125497 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:37.125498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:37.125507 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:37.125574 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:37.125792 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.126571 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.128263 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.128835 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.128919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.128931 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:37.129137 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:04:37.129183 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:04:37.129266 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.125406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:54785 (local address 127.31.115.195:41283)
0620 08:04:37.125455 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:37.125457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:37.125654 (+ 197us) server_negotiation.cc:407] Connection header received
0620 08:04:37.125735 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:37.125735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:37.125743 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:37.125820 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:37.125950 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.126626 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.128263 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.128835 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.128919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.128931 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:37.129137 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:04:37.129202 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:04:37.129276 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.125289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40327 (local address 127.31.115.194:40957)
0620 08:04:37.125362 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.125374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:37.125428 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:37.125622 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:37.125623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:37.125626 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:37.125713 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.125716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.126693 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.126695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.128138 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.128142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.128972 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.128973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.129037 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.129043 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:04:37.129136 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:04:37.129168 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":99}
I20260620 08:04:37.129282 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.125289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41283 (local address 127.31.115.194:54785)
0620 08:04:37.125573 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.125584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:37.125674 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:37.125840 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:37.125840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:37.125842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:37.125891 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.125892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.126703 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.126704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.128138 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:37.128142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.128947 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.128949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:37.129036 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.129044 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 08:04:37.129116 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:04:37.129168 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":144,"thread_start_us":112,"threads_started":1}
I20260620 08:04:37.129536 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ce7d1a8b7f45c6a99825e9735a04b5" candidate_uuid: "fe4e61e11f49480db968cbb18a9b438f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4afebc8d250f49698ee5741ed84d845c" is_pre_election: true
I20260620 08:04:37.129662 6779 raft_consensus.cc:2454] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe4e61e11f49480db968cbb18a9b438f in term 0.
I20260620 08:04:37.129637 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ce7d1a8b7f45c6a99825e9735a04b5" candidate_uuid: "fe4e61e11f49480db968cbb18a9b438f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" is_pre_election: true
I20260620 08:04:37.129792 7053 raft_consensus.cc:2454] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe4e61e11f49480db968cbb18a9b438f in term 0.
I20260620 08:04:37.129936 6851 leader_election.cc:302] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afebc8d250f49698ee5741ed84d845c, fe4e61e11f49480db968cbb18a9b438f; no voters:
I20260620 08:04:37.130060 7196 raft_consensus.cc:2781] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:37.130132 7196 raft_consensus.cc:482] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:37.130169 7196 raft_consensus.cc:3037] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:37.130878 7196 raft_consensus.cc:504] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.131042 7196 leader_election.cc:288] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [CANDIDATE]: Term 1 election: Requested vote from peers 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283), 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327)
I20260620 08:04:37.131217 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ce7d1a8b7f45c6a99825e9735a04b5" candidate_uuid: "fe4e61e11f49480db968cbb18a9b438f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4afebc8d250f49698ee5741ed84d845c"
I20260620 08:04:37.131292 6779 raft_consensus.cc:3037] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:37.131881 6779 raft_consensus.cc:2454] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe4e61e11f49480db968cbb18a9b438f in term 1.
I20260620 08:04:37.131986 7053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ce7d1a8b7f45c6a99825e9735a04b5" candidate_uuid: "fe4e61e11f49480db968cbb18a9b438f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3"
I20260620 08:04:37.132076 6851 leader_election.cc:302] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afebc8d250f49698ee5741ed84d845c, fe4e61e11f49480db968cbb18a9b438f; no voters:
I20260620 08:04:37.132097 7053 raft_consensus.cc:3037] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:37.132186 7196 raft_consensus.cc:2781] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:37.132401 7196 raft_consensus.cc:686] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 1 LEADER]: Becoming Leader. State: Replica: fe4e61e11f49480db968cbb18a9b438f, State: Running, Role: LEADER
I20260620 08:04:37.132485 7196 consensus_queue.cc:237] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } }
I20260620 08:04:37.132977 7053 raft_consensus.cc:2454] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe4e61e11f49480db968cbb18a9b438f in term 1.
I20260620 08:04:37.133317 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.133215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:54979)
0620 08:04:37.133259 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.133291 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 08:04:37.133364 6455 catalog_manager.cc:5495] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f reported cstate change: term changed from 0 to 1, leader changed from <none> to fe4e61e11f49480db968cbb18a9b438f (127.31.115.194). New cstate: current_term: 1 leader_uuid: "fe4e61e11f49480db968cbb18a9b438f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe4e61e11f49480db968cbb18a9b438f" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260620 08:04:37.143755 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.141332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.31.115.194:44645)
0620 08:04:37.141374 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:37.141377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:37.141385 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:37.141431 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:37.141432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:37.141453 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:37.141515 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:37.141672 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.142209 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.143031 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:37.143232 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:37.143284 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:37.143456 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:37.143558 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:37.143614 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:37.143663 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 08:04:37.143688 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:04:37.144824 6779 raft_consensus.cc:1261] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Refusing update from remote peer fe4e61e11f49480db968cbb18a9b438f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:37.144930 7053 raft_consensus.cc:1261] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Refusing update from remote peer fe4e61e11f49480db968cbb18a9b438f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:37.145180 7203 consensus_queue.cc:1035] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4afebc8d250f49698ee5741ed84d845c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:37.145331 7196 consensus_queue.cc:1035] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c8a779dc0b4df9af438cf8e2ca54a3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:37.146235 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.146144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.195:40675)
0620 08:04:37.146187 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.146208 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 08:04:37.146235 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.146118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:48851)
0620 08:04:37.146165 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.146201 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 08:04:37.147396 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.147302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.193:38301)
0620 08:04:37.147340 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.147367 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 08:04:37.147856 7206 mvcc.cc:205] Tried to move back new op lower bound from 7298837205583077376 to 7298837205535293440. Current Snapshot: MvccSnapshot[applied={T|T < 7298837205583077376}]
I20260620 08:04:37.148104 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.147997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.194:52195)
0620 08:04:37.148038 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.148073 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 08:04:37.148245 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.148149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34223 (local address 127.31.115.195:48233)
0620 08:04:37.148192 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.148216 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 08:04:37.150966 6455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:37.151047 6455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:37.152341 6455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e869551e011b4ea59efb71ecc8d0169e: Sending DeleteTablet for 3 replicas of tablet 74ce7d1a8b7f45c6a99825e9735a04b5
I20260620 08:04:37.152645 6759 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ce7d1a8b7f45c6a99825e9735a04b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:37 UTC) from {username='oryx', principal='oryx/127.31.115.253@KRBTEST.COM'} at 127.0.0.1:55540
I20260620 08:04:37.152865 7215 tablet_replica.cc:330] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: stopping tablet replica
I20260620 08:04:37.152930 7215 raft_consensus.cc:2229] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:37.152985 7215 raft_consensus.cc:2258] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:37.153020 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ce7d1a8b7f45c6a99825e9735a04b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:37 UTC) from {username='oryx', principal='oryx/127.31.115.253@KRBTEST.COM'} at 127.0.0.1:43076
I20260620 08:04:37.153205 7216 tablet_replica.cc:330] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: stopping tablet replica
I20260620 08:04:37.153264 7216 raft_consensus.cc:2229] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:37.153316 7216 raft_consensus.cc:2258] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:37.153330 6897 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ce7d1a8b7f45c6a99825e9735a04b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:37 UTC) from {username='oryx', principal='oryx/127.31.115.253@KRBTEST.COM'} at 127.0.0.1:53720
I20260620 08:04:37.153595 7215 ts_tablet_manager.cc:1905] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 4afebc8d250f49698ee5741ed84d845c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:37.153743 7216 ts_tablet_manager.cc:1905] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:37.153896 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6606
I20260620 08:04:37.155913 7216 ts_tablet_manager.cc:1918] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:37.156019 7216 log.cc:1196] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/74ce7d1a8b7f45c6a99825e9735a04b5
I20260620 08:04:37.156242 7216 ts_tablet_manager.cc:1939] T 74ce7d1a8b7f45c6a99825e9735a04b5 P 71c8a779dc0b4df9af438cf8e2ca54a3: Deleting consensus metadata
I20260620 08:04:37.153564 7217 tablet_replica.cc:330] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: stopping tablet replica
I20260620 08:04:37.156430 7217 raft_consensus.cc:2229] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:37.156551 7217 raft_consensus.cc:2258] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:37.156855 7217 ts_tablet_manager.cc:1905] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:37.156857 6442 catalog_manager.cc:4841] TS 71c8a779dc0b4df9af438cf8e2ca54a3 (127.31.115.195:41283): tablet 74ce7d1a8b7f45c6a99825e9735a04b5 (table test-table [id=7e372963c61041599cec6b93539a570a]) successfully deleted
I20260620 08:04:37.158282 7217 ts_tablet_manager.cc:1918] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:37.158360 7217 log.cc:1196] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/74ce7d1a8b7f45c6a99825e9735a04b5
I20260620 08:04:37.158604 7217 ts_tablet_manager.cc:1939] T 74ce7d1a8b7f45c6a99825e9735a04b5 P fe4e61e11f49480db968cbb18a9b438f: Deleting consensus metadata
I20260620 08:04:37.158989 6442 catalog_manager.cc:4841] TS fe4e61e11f49480db968cbb18a9b438f (127.31.115.194:44645): tablet 74ce7d1a8b7f45c6a99825e9735a04b5 (table test-table [id=7e372963c61041599cec6b93539a570a]) successfully deleted
I20260620 08:04:37.161273 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:37.161153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:40327 (local address 127.0.0.1:55552)
0620 08:04:37.161210 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:37.161241 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:40327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 08:04:37.161412 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6834
W20260620 08:04:37.161466 6440 catalog_manager.cc:4584] TS 4afebc8d250f49698ee5741ed84d845c (127.31.115.193:40327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74ce7d1a8b7f45c6a99825e9735a04b5: Network error: Client connection negotiation failed: client connection to 127.31.115.193:40327: connect: Connection refused (error 111)
I20260620 08:04:37.167547 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6970
I20260620 08:04:37.173969 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6322
I20260620 08:04:37.179275 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 6510
2026-06-20T08:04:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4689 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:37 dist-test-slave-c4w8 krb5kdc[7220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:37 dist-test-slave-c4w8 krb5kdc[7220](info): set up 2 sockets
Jun 20 08:04:37 dist-test-slave-c4w8 krb5kdc[7220](info): commencing operation
krb5kdc: starting...
W20260620 08:04:38.278734 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab.
Entry for principal oryx/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab.
I20260620 08:04:38.401666 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41825
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab
--principal=oryx/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:38.484694 7233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:38.484970 7233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:38.485009 7233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:38.486271 7233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:38.486352 7233 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:04:38.486373 7233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:38.486394 7233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:38.486428 7233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:38.486450 7233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:38.488035 7233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41825
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.254.keytab
--principal=oryx/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:38.488337 7233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:38.488584 7233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:38.493968 7239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:38.493963 7238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:38.493963 7241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:38.494194 7233 server_base.cc:1034] running on GCE node
I20260620 08:04:38.494457 7233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:38.494793 7233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:38.495973 7233 hybrid_clock.cc:648] HybridClock initialized: now 1781942678495960 us; error 31 us; skew 500 ppm
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:38.498716 7233 init.cc:377] Logged in from keytab as oryx/127.31.115.254@KRBTEST.COM (short username oryx)
I20260620 08:04:38.499185 7233 webserver.cc:460] Webserver started at http://127.31.115.254:42575/ using document root <none> and password file <none>
I20260620 08:04:38.499421 7233 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:38.499490 7233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:38.499639 7233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:38.500718 7233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "b98b0a7ea8074e9fb05a22123a0717af"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "ed82b5464478a49b5dd7439549e10bf6"
server_key_iv: "81205d85279da1a0c056c8da3b9c05bf"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.501087 7233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "b98b0a7ea8074e9fb05a22123a0717af"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "ed82b5464478a49b5dd7439549e10bf6"
server_key_iv: "81205d85279da1a0c056c8da3b9c05bf"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.502612 7233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:04:38.503482 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:38.503728 7233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:38.503844 7233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "b98b0a7ea8074e9fb05a22123a0717af"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "ed82b5464478a49b5dd7439549e10bf6"
server_key_iv: "81205d85279da1a0c056c8da3b9c05bf"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.503918 7233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:38.518620 7233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:38.520887 7233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:38.521081 7233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:38.526386 7233 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:41825
I20260620 08:04:38.526470 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:41825 every 8 connection(s)
I20260620 08:04:38.526876 7233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:38.527629 7311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:38.530205 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Bootstrap starting.
I20260620 08:04:38.530872 7311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:38.531200 7311 log.cc:824] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:38.531960 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: No bootstrap required, opened a new log
I20260620 08:04:38.534117 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98b0a7ea8074e9fb05a22123a0717af" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41825 } }
I20260620 08:04:38.534261 7311 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:38.534283 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98b0a7ea8074e9fb05a22123a0717af, State: Initialized, Role: FOLLOWER
I20260620 08:04:38.534467 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98b0a7ea8074e9fb05a22123a0717af" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41825 } }
I20260620 08:04:38.534543 7311 raft_consensus.cc:388] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:38.534565 7311 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:38.534653 7311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:38.535209 7311 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98b0a7ea8074e9fb05a22123a0717af" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41825 } }
I20260620 08:04:38.535297 7311 leader_election.cc:302] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b98b0a7ea8074e9fb05a22123a0717af; no voters:
I20260620 08:04:38.535602 7311 leader_election.cc:288] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:38.535655 7314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:38.535866 7314 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [term 1 LEADER]: Becoming Leader. State: Replica: b98b0a7ea8074e9fb05a22123a0717af, State: Running, Role: LEADER
I20260620 08:04:38.535925 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:38.536010 7314 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98b0a7ea8074e9fb05a22123a0717af" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41825 } }
I20260620 08:04:38.536389 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b98b0a7ea8074e9fb05a22123a0717af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98b0a7ea8074e9fb05a22123a0717af" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41825 } } }
I20260620 08:04:38.536489 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:38.536394 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98b0a7ea8074e9fb05a22123a0717af. Latest consensus state: current_term: 1 leader_uuid: "b98b0a7ea8074e9fb05a22123a0717af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98b0a7ea8074e9fb05a22123a0717af" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41825 } } }
I20260620 08:04:38.536638 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:38.536728 7322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:38.537094 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 7233
I20260620 08:04:38.537217 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:38.537338 7322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:38.537446 32207 external_mini_cluster.cc:1428] Setting key c7a89f6c6e528eb177fd69bf63cb21dc
I20260620 08:04:38.539495 7322 catalog_manager.cc:1347] Generated new cluster ID: 23d2edeab9bc42529955f6ab92ecb09a
I20260620 08:04:38.539553 7322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:38.549683 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.537973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.31.115.254:41825)
0620 08:04:38.538177 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:38.538187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:38.538221 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:04:38.538355 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:38.538361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:38.538417 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:38.538570 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:38.539348 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.540758 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.541873 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.542504 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.542603 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.545548 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:38.545553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:38.545554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:38.545569 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:38.547686 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.548151 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.548156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.548156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.548283 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.548495 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.548495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.548496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.548854 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:38.549017 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:38.549127 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:04:38.549389 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260620 08:04:38.551244 7322 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:38.552013 7322 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:38.562563 7322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Generated new TSK 0
I20260620 08:04:38.562803 7322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab.
Entry for principal oryx/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab.
I20260620 08:04:38.597316 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41825
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:38.681488 7337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:38.681808 7337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:38.681857 7337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:38.683094 7337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:38.683177 7337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:38.683233 7337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:38.683272 7337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:38.684899 7337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:38.685211 7337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:38.685477 7337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:38.690937 7343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:38.691039 7342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:38.691105 7337 server_base.cc:1034] running on GCE node
W20260620 08:04:38.691049 7345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:38.691375 7337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:38.691622 7337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:38.692914 7337 hybrid_clock.cc:648] HybridClock initialized: now 1781942678692852 us; error 47 us; skew 500 ppm
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:38.698727 7337 init.cc:377] Logged in from keytab as oryx/127.31.115.193@KRBTEST.COM (short username oryx)
I20260620 08:04:38.699244 7337 webserver.cc:460] Webserver started at http://127.31.115.193:36977/ using document root <none> and password file <none>
I20260620 08:04:38.699450 7337 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:38.699491 7337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:38.699615 7337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:38.700753 7337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "ec37b82c16a1474caf037a1d98a5f9b5"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "e2d2461bfb7923255158b5bdd1681ca9"
server_key_iv: "3fae70de07da9de2dfd3a0a52f582a61"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.701052 7337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "ec37b82c16a1474caf037a1d98a5f9b5"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "e2d2461bfb7923255158b5bdd1681ca9"
server_key_iv: "3fae70de07da9de2dfd3a0a52f582a61"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.702904 7337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 08:04:38.703796 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:38.704095 7337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:38.704197 7337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "ec37b82c16a1474caf037a1d98a5f9b5"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "e2d2461bfb7923255158b5bdd1681ca9"
server_key_iv: "3fae70de07da9de2dfd3a0a52f582a61"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.704262 7337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:38.719763 7337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:38.722182 7337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:38.722307 7337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:38.722916 7337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:38.722963 7337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:38.722990 7337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:38.723004 7337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:38.730029 7337 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:35453
I20260620 08:04:38.730154 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:35453 every 8 connection(s)
I20260620 08:04:38.730449 7337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:38.733104 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 7337
I20260620 08:04:38.733194 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:38.733458 32207 external_mini_cluster.cc:1428] Setting key c8f86c31d153090f7b729f97fb423683
I20260620 08:04:38.734512 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.723432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:40373 (local address 127.31.115.254:41825)
0620 08:04:38.723519 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:38.723522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:38.723757 (+ 235us) server_negotiation.cc:407] Connection header received
0620 08:04:38.724276 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:38.724278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:38.724297 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:38.724367 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:38.725389 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.726310 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.727945 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.728312 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.728398 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.731476 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:38.731478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:38.731478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:38.731485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:38.732781 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.733254 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.733256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.733256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.733324 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.733693 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.733694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.733694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.733864 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:38.733961 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:38.734321 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 08:04:38.734423 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:04:38.734692 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.723403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.31.115.193:40373)
0620 08:04:38.723606 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:38.723648 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:38.724138 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:38.724457 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:38.724462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:38.724952 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:38.725268 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.725275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.726417 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.726418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.727845 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.727848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.728460 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.728461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.728664 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.729475 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:38.729484 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:38.731382 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:38.732859 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.732866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.732868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.733115 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.733401 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.733402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.733402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.733538 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.733949 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:38.733957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:38.734174 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 08:04:38.734285 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:04:38.734450 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260620 08:04:38.736388 7468 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41825
I20260620 08:04:38.736482 7468 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:38.736675 7468 heartbeater.cc:511] Master 127.31.115.254:41825 requested a full tablet report, sending...
I20260620 08:04:38.737268 7264 ts_manager.cc:194] Registered new tserver with Master: ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453)
I20260620 08:04:38.738006 7264 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:40373
WARNING: no policy specified for oryx/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:04:38.743734 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.736409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.31.115.193:41771)
0620 08:04:38.736621 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:38.736632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:38.736756 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:38.736941 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:38.736942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:38.737203 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:38.737285 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.737288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.738341 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.738342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.739331 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.739335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.739761 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.739762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.739845 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.740313 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:38.740319 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:38.740836 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:38.742633 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.742635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.742635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.742859 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.743078 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.743079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.743079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.743164 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.743436 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:38.743437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:38.743504 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 08:04:38.743570 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:04:38.743654 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260620 08:04:38.743782 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.736449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:41771 (local address 127.31.115.254:41825)
0620 08:04:38.736488 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:38.736490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:38.736703 (+ 213us) server_negotiation.cc:407] Connection header received
0620 08:04:38.736857 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:38.736859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:38.736873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:38.736936 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:38.737377 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.738214 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.739449 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.739669 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.739735 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.740940 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:38.740942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:38.740942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:38.740949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:38.742537 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.742948 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.742949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.742950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.743002 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.743241 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.743242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.743242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.743362 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:38.743426 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:38.743635 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 08:04:38.743716 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab.
Entry for principal oryx/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab.
I20260620 08:04:38.761080 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41825
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:38.847177 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:38.847493 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:38.847524 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:38.848933 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:38.848996 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:38.849020 7477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:38.849040 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:38.850821 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:38.851099 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:38.851358 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:04:38.856642 7477 server_base.cc:1034] running on GCE node
W20260620 08:04:38.856593 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:38.856740 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:38.856658 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:38.857017 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:38.857340 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:38.858490 7477 hybrid_clock.cc:648] HybridClock initialized: now 1781942678858468 us; error 37 us; skew 500 ppm
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:38.861109 7477 init.cc:377] Logged in from keytab as oryx/127.31.115.194@KRBTEST.COM (short username oryx)
I20260620 08:04:38.861685 7477 webserver.cc:460] Webserver started at http://127.31.115.194:45073/ using document root <none> and password file <none>
I20260620 08:04:38.861912 7477 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:38.861979 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:38.862139 7477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:38.863253 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "6f71c011e44c9926cdb9c8d238126e8e"
server_key_iv: "25bc39befe9a39fe637760666f961b87"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.863624 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "6f71c011e44c9926cdb9c8d238126e8e"
server_key_iv: "25bc39befe9a39fe637760666f961b87"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.865046 7477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:38.866035 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:38.866272 7477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:38.866369 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169"
format_stamp: "Formatted at 2026-06-20 08:04:38 on dist-test-slave-c4w8"
server_key: "6f71c011e44c9926cdb9c8d238126e8e"
server_key_iv: "25bc39befe9a39fe637760666f961b87"
server_key_version: "encryptionkey@0"
I20260620 08:04:38.866427 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:38.883185 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:38.885882 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:38.886031 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:38.886713 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:38.886765 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:38.886801 7477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:38.886819 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:04:38 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:38.894415 7477 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:34935
I20260620 08:04:38.894464 7607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:34935 every 8 connection(s)
I20260620 08:04:38.894804 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:38.896811 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 7477
I20260620 08:04:38.896915 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:38.897171 32207 external_mini_cluster.cc:1428] Setting key 455bea3bce66b30ce793e2f8123844a4
I20260620 08:04:38.898118 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.887148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:34465 (local address 127.31.115.254:41825)
0620 08:04:38.887200 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:38.887202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:38.887534 (+ 332us) server_negotiation.cc:407] Connection header received
0620 08:04:38.888326 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:38.888328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:38.888343 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:38.888407 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:38.889381 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.890164 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.891907 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.892247 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.892316 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.894784 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:38.894786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:38.894786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:38.894794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:38.896284 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.896825 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.896826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.896827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.896932 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.897302 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.897303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.897304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.897476 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:38.897570 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:38.897903 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 08:04:38.898001 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:04:38.898342 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.887157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.31.115.194:34465)
0620 08:04:38.887357 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:38.887416 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:38.888164 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:38.888499 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:38.888504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:38.888935 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:38.889265 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.889273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.890283 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.890285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.891776 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.891781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.892327 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.892328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.892469 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.893045 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:38.893055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:38.894676 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:38.896398 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.896406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.896408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.896706 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.897045 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.897046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.897046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.897198 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.897557 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:38.897564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:38.897761 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 08:04:38.897852 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:04:38.898068 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260620 08:04:38.898978 7608 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41825
I20260620 08:04:38.899091 7608 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:38.899308 7608 heartbeater.cc:511] Master 127.31.115.254:41825 requested a full tablet report, sending...
I20260620 08:04:38.899783 7265 ts_manager.cc:194] Registered new tserver with Master: 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935)
I20260620 08:04:38.900336 7265 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:34465
I20260620 08:04:38.907114 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.900150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:55561 (local address 127.31.115.254:41825)
0620 08:04:38.900191 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:38.900193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:38.900361 (+ 168us) server_negotiation.cc:407] Connection header received
0620 08:04:38.900431 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:38.900431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:38.900443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:38.900472 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:38.900897 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.901597 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.902347 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.902678 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.902757 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.903760 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:38.903761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:38.903762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:38.903770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:38.905446 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.906024 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.906025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.906026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.906090 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:38.906422 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:38.906423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:38.906423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:38.906685 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:38.906759 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:38.906914 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:04:38.907000 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:04:38.907132 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:38.900104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.31.115.194:55561)
0620 08:04:38.900287 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:38.900298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:38.900361 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:38.900536 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:38.900537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:38.900751 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:38.900817 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.900818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.901726 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.901727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.902236 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:38.902238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:38.902764 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:38.902765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:38.902819 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:38.903322 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:38.903326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:38.903663 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:38.905730 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.905731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.905732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.905939 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.906212 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:38.906213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:38.906214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:38.906339 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:38.906765 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:38.906766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:38.906813 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:04:38.906883 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:04:38.906983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for oryx/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab.
Entry for principal oryx/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab.
I20260620 08:04:38.926930 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41825
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:39.014104 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:39.014336 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:39.014361 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:39.015547 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:39.015590 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:39.015612 7617 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:04:39.015636 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:39.017182 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.115.254:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:39.017390 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:39.017584 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:39.022873 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:39.022861 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:39.022861 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:39.023236 7617 server_base.cc:1034] running on GCE node
I20260620 08:04:39.023473 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:39.023752 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:39.024946 7617 hybrid_clock.cc:648] HybridClock initialized: now 1781942679024897 us; error 68 us; skew 500 ppm
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942679, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:39.027947 7617 init.cc:377] Logged in from keytab as oryx/127.31.115.195@KRBTEST.COM (short username oryx)
I20260620 08:04:39.028445 7617 webserver.cc:460] Webserver started at http://127.31.115.195:37869/ using document root <none> and password file <none>
I20260620 08:04:39.028694 7617 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:39.028767 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:39.028915 7617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:39.030090 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "bb77008afb704765bced49ed988d305a"
format_stamp: "Formatted at 2026-06-20 08:04:39 on dist-test-slave-c4w8"
server_key: "65ae97c85194add23428d8ac5fadf4d7"
server_key_iv: "7eba7c582cad77a83490dd1c5a7ca570"
server_key_version: "encryptionkey@0"
I20260620 08:04:39.030462 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "bb77008afb704765bced49ed988d305a"
format_stamp: "Formatted at 2026-06-20 08:04:39 on dist-test-slave-c4w8"
server_key: "65ae97c85194add23428d8ac5fadf4d7"
server_key_iv: "7eba7c582cad77a83490dd1c5a7ca570"
server_key_version: "encryptionkey@0"
I20260620 08:04:39.031945 7617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:39.032884 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:39.033068 7617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:39.033195 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "bb77008afb704765bced49ed988d305a"
format_stamp: "Formatted at 2026-06-20 08:04:39 on dist-test-slave-c4w8"
server_key: "65ae97c85194add23428d8ac5fadf4d7"
server_key_iv: "7eba7c582cad77a83490dd1c5a7ca570"
server_key_version: "encryptionkey@0"
I20260620 08:04:39.033288 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:39.064097 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:39.066745 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:39.066920 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:39.067462 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:39.067528 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:39.067577 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:39.067610 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942679, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:39.075016 7617 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:45727
I20260620 08:04:39.075099 7747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:45727 every 8 connection(s)
I20260620 08:04:39.075464 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:39.077899 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.068046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:56083 (local address 127.31.115.254:41825)
0620 08:04:39.068090 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.068092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.068312 (+ 220us) server_negotiation.cc:407] Connection header received
0620 08:04:39.068900 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.068901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.068915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.068975 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.069875 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.070605 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.071716 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.071932 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.072012 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.074880 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:39.074883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:39.074883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:39.074893 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:39.076419 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.076829 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.076831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.076831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.076902 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.077173 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.077174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.077174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.077289 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:39.077355 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.077731 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.077832 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:04:39.078034 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.068020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.31.115.195:56083)
0620 08:04:39.068191 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.068226 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.068783 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.069056 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.069080 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.069439 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:39.069771 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.069777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.070699 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.070700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.071619 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.071622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.072032 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.072033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.072190 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.072813 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:39.072826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:39.074763 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:39.076494 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:39.076498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:39.076500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:39.076741 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:39.076991 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:39.076992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:39.076992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:39.077102 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:39.077365 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:39.077370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:39.077494 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.077587 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.077829 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260620 08:04:39.078554 7748 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41825
I20260620 08:04:39.078646 7748 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:39.078833 7748 heartbeater.cc:511] Master 127.31.115.254:41825 requested a full tablet report, sending...
I20260620 08:04:39.079210 7265 ts_manager.cc:194] Registered new tserver with Master: bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:39.079718 7265 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:56083
I20260620 08:04:39.082751 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 7617
I20260620 08:04:39.082856 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:39.083079 32207 external_mini_cluster.cc:1428] Setting key 4f84bde27bbe87f81e02f2867587defd
I20260620 08:04:39.084163 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:04:39.086239 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.079760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:58725 (local address 127.31.115.254:41825)
0620 08:04:39.079802 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.079804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.079878 (+ 74us) server_negotiation.cc:407] Connection header received
0620 08:04:39.079964 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.079965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.079974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.080019 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.080777 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.081241 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.081961 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.082151 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.082204 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.083268 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:39.083269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:39.083270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:39.083277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:39.084924 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.085356 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.085357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.085357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.085420 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.085718 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.085719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.085719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.085844 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:39.085899 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.086085 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.086150 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:04:39.086248 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.079662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.31.115.195:58725)
0620 08:04:39.079810 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.079822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.079905 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.080501 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.080502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.080654 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:39.080699 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.080701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.081328 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.081329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.081896 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.081898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.082215 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.082216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.082261 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.082684 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:39.082690 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:39.083082 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:39.085008 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:39.085009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:39.085009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:39.085247 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:39.085504 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:39.085504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:39.085505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:39.085578 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:39.085915 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:39.085916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:39.085973 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.086061 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.086181 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:04:39.107124 7333 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:39.101262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.31.115.254:41825)
0620 08:04:39.101330 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.101332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.101342 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:39.101393 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.101394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.101403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.101456 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.102112 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.102678 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.103476 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.103668 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.103732 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.107003 (+ 3271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57214: BlockingRecv error: recv got EOF from 127.0.0.1:57214 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:39.115617 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.107984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.31.115.254:41825)
0620 08:04:39.108074 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.108078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.108092 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:04:39.108168 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.108168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.108177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.108226 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.108727 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.109184 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.110052 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.110243 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.110300 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.112604 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:39.112606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:39.112606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:39.112614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:39.114330 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.114781 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.114783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.114783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.114858 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.115140 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.115141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.115142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.115282 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:39.115374 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.115477 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.115545 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:04:39.117480 7265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:39.119182 7265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:39.127391 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.122002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.31.115.193:35453)
0620 08:04:39.122777 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.122782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.122792 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:39.122834 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.122837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.122853 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.122954 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.123170 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.124296 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.125933 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.126809 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.126935 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.127013 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.127235 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.127279 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":150,"threads_started":1}
I20260620 08:04:39.127380 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.121855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.0.0.1:59306)
0620 08:04:39.121986 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.122029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.122093 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.122955 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.122956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.122958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.123068 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.123075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.124414 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.124415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.125822 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.125826 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.126994 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.126996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.127091 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.127103 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.127209 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.127273 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":72,"thread_start_us":75,"threads_started":1}
I20260620 08:04:39.127843 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.122083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34935 (local address 127.0.0.1:50230)
0620 08:04:39.122367 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.122379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.122437 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.122959 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.122960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.122962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.123055 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.123058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.124412 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.124413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.126491 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.126494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.127616 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.127617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.127688 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.127692 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.127729 (+ 37us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.127769 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
I20260620 08:04:39.128172 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.122400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.31.115.194:34935)
0620 08:04:39.122737 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.122742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.122753 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:39.122793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.122795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.122812 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.122886 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.123144 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.124301 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.126637 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.127399 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.127947 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.128013 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.128041 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.128080 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":89,"threads_started":1}
I20260620 08:04:39.128295 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1615bb475440f4aefec43e0b64de39 (DEFAULT_TABLE table=test-table [id=ded6a0f8afea4d8fa91b44a444be740d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.128710 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1615bb475440f4aefec43e0b64de39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.128819 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1615bb475440f4aefec43e0b64de39 (DEFAULT_TABLE table=test-table [id=ded6a0f8afea4d8fa91b44a444be740d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.129200 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1615bb475440f4aefec43e0b64de39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.132068 7777 tablet_bootstrap.cc:492] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Bootstrap starting.
I20260620 08:04:39.132622 7778 tablet_bootstrap.cc:492] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Bootstrap starting.
I20260620 08:04:39.132936 7777 tablet_bootstrap.cc:654] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.133080 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.122259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:45727 (local address 127.0.0.1:46278)
0620 08:04:39.122586 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.122593 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.122634 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.122940 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.122944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.122947 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.123055 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.123058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.125397 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.125399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.126622 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.126625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.132794 (+ 6169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.132796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.132876 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.132883 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.132950 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.133006 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260620 08:04:39.133172 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.122359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.31.115.195:45727)
0620 08:04:39.122536 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.122542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.122652 (+ 110us) server_negotiation.cc:407] Connection header received
0620 08:04:39.122715 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.122718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.122737 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.122838 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.124414 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.125314 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.126725 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.132587 (+ 5862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.132690 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.132751 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.133003 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.133066 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":167,"threads_started":1}
I20260620 08:04:39.133683 7777 log.cc:824] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:39.133834 7778 tablet_bootstrap.cc:654] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.133899 7682 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1615bb475440f4aefec43e0b64de39 (DEFAULT_TABLE table=test-table [id=ded6a0f8afea4d8fa91b44a444be740d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.134187 7778 log.cc:824] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:39.134289 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1615bb475440f4aefec43e0b64de39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.134609 7777 tablet_bootstrap.cc:492] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: No bootstrap required, opened a new log
I20260620 08:04:39.134680 7777 ts_tablet_manager.cc:1397] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 08:04:39.135023 7778 tablet_bootstrap.cc:492] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: No bootstrap required, opened a new log
I20260620 08:04:39.135116 7778 ts_tablet_manager.cc:1397] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 08:04:39.136837 7777 raft_consensus.cc:348] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.137048 7777 raft_consensus.cc:374] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.137089 7777 raft_consensus.cc:729] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.137274 7777 consensus_queue.cc:260] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.137382 7778 raft_consensus.cc:348] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.137545 7778 raft_consensus.cc:374] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.137589 7778 raft_consensus.cc:729] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.137578 7777 ts_tablet_manager.cc:1428] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:04:39.137748 7468 heartbeater.cc:503] Master 127.31.115.254:41825 was elected leader, sending a full tablet report...
I20260620 08:04:39.137753 7778 consensus_queue.cc:260] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.138234 7782 tablet_bootstrap.cc:492] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Bootstrap starting.
I20260620 08:04:39.138628 7608 heartbeater.cc:503] Master 127.31.115.254:41825 was elected leader, sending a full tablet report...
I20260620 08:04:39.138628 7778 ts_tablet_manager.cc:1428] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 08:04:39.138952 7782 tablet_bootstrap.cc:654] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.139294 7782 log.cc:824] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:39.140003 7782 tablet_bootstrap.cc:492] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: No bootstrap required, opened a new log
I20260620 08:04:39.140092 7782 ts_tablet_manager.cc:1397] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:39.141450 7782 raft_consensus.cc:348] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.141600 7782 raft_consensus.cc:374] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.141664 7782 raft_consensus.cc:729] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb77008afb704765bced49ed988d305a, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.141782 7782 consensus_queue.cc:260] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.142021 7782 ts_tablet_manager.cc:1428] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:04:39.142082 7748 heartbeater.cc:503] Master 127.31.115.254:41825 was elected leader, sending a full tablet report...
W20260620 08:04:39.145403 7609 tablet.cc:2369] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:39.231333 7469 tablet.cc:2369] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:04:39.326354 7749 tablet.cc:2369] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:04:39.376219 7781 raft_consensus.cc:482] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:39.376412 7781 raft_consensus.cc:504] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.376786 7781 leader_election.cc:288] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:39.380061 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.377228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:45727 (local address 127.31.115.193:41763)
0620 08:04:39.377337 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.377348 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.377399 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.377539 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.377558 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.377560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.377667 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.377669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.378380 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.378381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.379218 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.379221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.379788 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.379789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.379866 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.379869 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.379937 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.379978 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":43,"threads_started":1}
I20260620 08:04:39.380051 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.377327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:41763 (local address 127.31.115.195:45727)
0620 08:04:39.377369 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.377371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.377383 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:04:39.377452 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.377453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.377463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.377521 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.377739 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.378288 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.379308 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.379689 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.379785 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.379795 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.379944 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.379980 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:04:39.380152 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.377127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:48591 (local address 127.31.115.194:34935)
0620 08:04:39.377185 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.377188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.377221 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:04:39.377286 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.377288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.377300 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.377365 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.377586 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.378225 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.379317 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.379784 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.379880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.379889 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.380044 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.380078 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:04:39.380358 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.377006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34935 (local address 127.31.115.193:48591)
0620 08:04:39.377133 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.377146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.377204 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.377391 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.377392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.377395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.377488 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.377491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.378323 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.378324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.379230 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.379233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.379883 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.379884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.379957 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.379961 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.380036 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.380078 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260620 08:04:39.380582 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1615bb475440f4aefec43e0b64de39" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a" is_pre_election: true
I20260620 08:04:39.380748 7702 raft_consensus.cc:2454] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 0.
I20260620 08:04:39.380704 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1615bb475440f4aefec43e0b64de39" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" is_pre_election: true
I20260620 08:04:39.380853 7562 raft_consensus.cc:2454] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 0.
I20260620 08:04:39.380962 7355 leader_election.cc:302] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb77008afb704765bced49ed988d305a, ec37b82c16a1474caf037a1d98a5f9b5; no voters:
I20260620 08:04:39.381096 7781 raft_consensus.cc:2781] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:39.381161 7781 raft_consensus.cc:482] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:39.381187 7781 raft_consensus.cc:3037] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.381870 7781 raft_consensus.cc:504] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.382009 7781 leader_election.cc:288] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:39.382179 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1615bb475440f4aefec43e0b64de39" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169"
I20260620 08:04:39.382189 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1615bb475440f4aefec43e0b64de39" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a"
I20260620 08:04:39.382261 7562 raft_consensus.cc:3037] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.382261 7702 raft_consensus.cc:3037] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.383106 7702 raft_consensus.cc:2454] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 1.
I20260620 08:04:39.383106 7562 raft_consensus.cc:2454] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 1.
I20260620 08:04:39.383358 7355 leader_election.cc:302] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb77008afb704765bced49ed988d305a, ec37b82c16a1474caf037a1d98a5f9b5; no voters:
I20260620 08:04:39.383458 7781 raft_consensus.cc:2781] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:39.383582 7781 raft_consensus.cc:686] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 LEADER]: Becoming Leader. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Running, Role: LEADER
I20260620 08:04:39.383662 7781 consensus_queue.cc:237] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.384536 7264 catalog_manager.cc:5495] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:39.411249 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.408402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.31.115.193:35453)
0620 08:04:39.408456 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.408459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.408470 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:39.408588 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.408589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.408616 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.408705 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:39.408868 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.409388 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.410362 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.410584 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.410662 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.410880 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.411003 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.411108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.411130 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.411152 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:39.413151 7562 raft_consensus.cc:1261] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Refusing update from remote peer ec37b82c16a1474caf037a1d98a5f9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:39.413184 7702 raft_consensus.cc:1261] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Refusing update from remote peer ec37b82c16a1474caf037a1d98a5f9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:39.413648 7788 consensus_queue.cc:1035] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:39.413802 7781 consensus_queue.cc:1035] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:39.417377 7792 mvcc.cc:205] Tried to move back new op lower bound from 7298837214873354240 to 7298837214756073472. Current Snapshot: MvccSnapshot[applied={T|T < 7298837214873354240}]
I20260620 08:04:39.419137 7265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:39.419250 7265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:39.420059 7265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Sending DeleteTablet for 3 replicas of tablet 0f1615bb475440f4aefec43e0b64de39
I20260620 08:04:39.420414 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1615bb475440f4aefec43e0b64de39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:39 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59306
I20260620 08:04:39.420680 7800 tablet_replica.cc:330] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: stopping tablet replica
I20260620 08:04:39.420617 7265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:39.420768 7265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:39.420737 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1615bb475440f4aefec43e0b64de39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:39 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46278
I20260620 08:04:39.420782 7800 raft_consensus.cc:2229] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:39.420874 7800 raft_consensus.cc:2258] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:39.421027 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1615bb475440f4aefec43e0b64de39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:39 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50230
I20260620 08:04:39.421232 7802 tablet_replica.cc:330] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: stopping tablet replica
I20260620 08:04:39.421356 7802 raft_consensus.cc:2229] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:39.421427 7800 ts_tablet_manager.cc:1905] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:39.421583 7802 raft_consensus.cc:2258] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:39.422077 7802 ts_tablet_manager.cc:1905] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:39.423051 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 58a8b9090316497581a5dba0e00111fe (DEFAULT_TABLE table=test-table [id=c7d9ba26a2c54992bb1fccf8573d6abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.423192 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a8b9090316497581a5dba0e00111fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.423722 7800 ts_tablet_manager.cc:1918] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:39.423804 7800 log.cc:1196] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/0f1615bb475440f4aefec43e0b64de39
I20260620 08:04:39.424060 7802 ts_tablet_manager.cc:1918] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:39.424196 7802 log.cc:1196] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/0f1615bb475440f4aefec43e0b64de39
I20260620 08:04:39.424413 7800 ts_tablet_manager.cc:1939] T 0f1615bb475440f4aefec43e0b64de39 P ec37b82c16a1474caf037a1d98a5f9b5: Deleting consensus metadata
I20260620 08:04:39.424566 7778 tablet_bootstrap.cc:492] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Bootstrap starting.
I20260620 08:04:39.424650 7682 tablet_service.cc:1467] Processing CreateTablet for tablet 58a8b9090316497581a5dba0e00111fe (DEFAULT_TABLE table=test-table [id=c7d9ba26a2c54992bb1fccf8573d6abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.424779 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a8b9090316497581a5dba0e00111fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.424816 7251 catalog_manager.cc:4841] TS ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453): tablet 0f1615bb475440f4aefec43e0b64de39 (table test-table [id=ded6a0f8afea4d8fa91b44a444be740d]) successfully deleted
I20260620 08:04:39.424953 7778 tablet_bootstrap.cc:654] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.425386 7802 ts_tablet_manager.cc:1939] T 0f1615bb475440f4aefec43e0b64de39 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting consensus metadata
I20260620 08:04:39.425474 7778 tablet_bootstrap.cc:492] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: No bootstrap required, opened a new log
I20260620 08:04:39.425523 7778 ts_tablet_manager.cc:1397] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:04:39.425900 7778 raft_consensus.cc:348] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.426002 7778 raft_consensus.cc:374] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.426033 7778 raft_consensus.cc:729] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.426076 7778 consensus_queue.cc:260] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.426190 7778 ts_tablet_manager.cc:1428] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:39.426432 7782 tablet_bootstrap.cc:492] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Bootstrap starting.
I20260620 08:04:39.426878 7782 tablet_bootstrap.cc:654] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.427451 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 58a8b9090316497581a5dba0e00111fe (DEFAULT_TABLE table=test-table [id=c7d9ba26a2c54992bb1fccf8573d6abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.427587 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a8b9090316497581a5dba0e00111fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.427640 7782 tablet_bootstrap.cc:492] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: No bootstrap required, opened a new log
I20260620 08:04:39.427685 7782 ts_tablet_manager.cc:1397] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:39.427845 7782 raft_consensus.cc:348] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.427920 7782 raft_consensus.cc:374] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.427947 7782 raft_consensus.cc:729] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb77008afb704765bced49ed988d305a, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.427995 7782 consensus_queue.cc:260] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.428089 7782 ts_tablet_manager.cc:1428] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:39.428957 7777 tablet_bootstrap.cc:492] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Bootstrap starting.
I20260620 08:04:39.429378 7777 tablet_bootstrap.cc:654] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.429813 7251 catalog_manager.cc:4841] TS 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935): tablet 0f1615bb475440f4aefec43e0b64de39 (table test-table [id=ded6a0f8afea4d8fa91b44a444be740d]) successfully deleted
I20260620 08:04:39.430047 7801 tablet_replica.cc:330] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: stopping tablet replica
I20260620 08:04:39.430184 7801 raft_consensus.cc:2229] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:39.430325 7801 raft_consensus.cc:2258] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:39.430835 7801 ts_tablet_manager.cc:1905] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:39.430860 7777 tablet_bootstrap.cc:492] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: No bootstrap required, opened a new log
I20260620 08:04:39.430909 7777 ts_tablet_manager.cc:1397] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:04:39.431075 7777 raft_consensus.cc:348] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.431171 7777 raft_consensus.cc:374] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.431196 7777 raft_consensus.cc:729] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.431245 7777 consensus_queue.cc:260] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.431348 7777 ts_tablet_manager.cc:1428] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:39.432588 7801 ts_tablet_manager.cc:1918] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:39.432677 7801 log.cc:1196] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/0f1615bb475440f4aefec43e0b64de39
I20260620 08:04:39.432909 7801 ts_tablet_manager.cc:1939] T 0f1615bb475440f4aefec43e0b64de39 P bb77008afb704765bced49ed988d305a: Deleting consensus metadata
I20260620 08:04:39.433336 7251 catalog_manager.cc:4841] TS bb77008afb704765bced49ed988d305a (127.31.115.195:45727): tablet 0f1615bb475440f4aefec43e0b64de39 (table test-table [id=ded6a0f8afea4d8fa91b44a444be740d]) successfully deleted
I20260620 08:04:39.603853 7785 raft_consensus.cc:482] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:39.604007 7785 raft_consensus.cc:504] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.604385 7785 leader_election.cc:288] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453), 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935)
I20260620 08:04:39.607805 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.604963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:60295 (local address 127.31.115.194:34935)
0620 08:04:39.605027 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.605029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.605039 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:39.605107 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.605108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.605119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.605201 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.605430 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.606034 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.607090 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.607440 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.607532 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.607542 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.607682 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.607716 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:04:39.607810 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.604854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34935 (local address 127.31.115.195:60295)
0620 08:04:39.604982 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.604991 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.605035 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.605249 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.605249 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.605254 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.605315 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.605317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.606118 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.606119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.607006 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.607008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.607532 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.607533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.607615 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.607620 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.607671 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.607714 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260620 08:04:39.607954 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.604637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.31.115.195:42685)
0620 08:04:39.604781 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.604794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.604850 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.605027 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.605029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.605032 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:39.605114 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.605116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.605946 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.605947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.606980 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.606983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.607700 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.607701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.607776 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.607781 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.607833 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.607874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260620 08:04:39.607954 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.604753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:42685 (local address 127.31.115.193:35453)
0620 08:04:39.604851 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.604853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.604908 (+ 55us) server_negotiation.cc:407] Connection header received
0620 08:04:39.604940 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.604941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.604955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.605008 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:39.605173 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.605861 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.607066 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.607618 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.607700 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.607709 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.607838 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.607876 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:04:39.608060 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a8b9090316497581a5dba0e00111fe" candidate_uuid: "bb77008afb704765bced49ed988d305a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" is_pre_election: true
I20260620 08:04:39.608173 7562 raft_consensus.cc:2454] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb77008afb704765bced49ed988d305a in term 0.
I20260620 08:04:39.608248 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a8b9090316497581a5dba0e00111fe" candidate_uuid: "bb77008afb704765bced49ed988d305a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" is_pre_election: true
I20260620 08:04:39.608350 7635 leader_election.cc:302] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, bb77008afb704765bced49ed988d305a; no voters:
I20260620 08:04:39.608397 7422 raft_consensus.cc:2454] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb77008afb704765bced49ed988d305a in term 0.
I20260620 08:04:39.608472 7785 raft_consensus.cc:2781] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:39.608510 7785 raft_consensus.cc:482] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:39.608531 7785 raft_consensus.cc:3037] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.609220 7785 raft_consensus.cc:504] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.609362 7785 leader_election.cc:288] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [CANDIDATE]: Term 1 election: Requested vote from peers ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453), 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935)
I20260620 08:04:39.609513 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a8b9090316497581a5dba0e00111fe" candidate_uuid: "bb77008afb704765bced49ed988d305a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec37b82c16a1474caf037a1d98a5f9b5"
I20260620 08:04:39.609546 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a8b9090316497581a5dba0e00111fe" candidate_uuid: "bb77008afb704765bced49ed988d305a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169"
I20260620 08:04:39.609596 7422 raft_consensus.cc:3037] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.609637 7562 raft_consensus.cc:3037] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.610365 7562 raft_consensus.cc:2454] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb77008afb704765bced49ed988d305a in term 1.
I20260620 08:04:39.610487 7422 raft_consensus.cc:2454] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb77008afb704765bced49ed988d305a in term 1.
I20260620 08:04:39.610529 7635 leader_election.cc:302] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, bb77008afb704765bced49ed988d305a; no voters:
I20260620 08:04:39.610651 7785 raft_consensus.cc:2781] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:39.610813 7785 raft_consensus.cc:686] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 1 LEADER]: Becoming Leader. State: Replica: bb77008afb704765bced49ed988d305a, State: Running, Role: LEADER
I20260620 08:04:39.610922 7785 consensus_queue.cc:237] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.611686 7265 catalog_manager.cc:5495] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb77008afb704765bced49ed988d305a (127.31.115.195). New cstate: current_term: 1 leader_uuid: "bb77008afb704765bced49ed988d305a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } health_report { overall_health: HEALTHY } } }
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942678, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.254@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:39.682708 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.674436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41825 (local address 127.0.0.1:57234)
0620 08:04:39.674715 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.674725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.674801 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.674993 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.674994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.675493 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:39.675550 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.675553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.676347 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.676348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.676962 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.676964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.677377 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.677379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.677431 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.678221 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:39.678226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:39.679794 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:39.681509 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:39.681518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:39.681521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:39.681831 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:39.682076 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:39.682077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:39.682077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:39.682137 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:39.682373 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:39.682379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:39.682426 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.682477 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.682592 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":4,"thread_start_us":178,"threads_started":1}
I20260620 08:04:39.682719 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.674529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.31.115.254:41825)
0620 08:04:39.674712 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.674714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.674824 (+ 110us) server_negotiation.cc:407] Connection header received
0620 08:04:39.674858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.674859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.674873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.674978 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.675647 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.676230 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.677082 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.677280 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.677347 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.679888 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:39.679889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:39.679889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:39.679896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:39.681420 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.681932 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.681933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.681933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.682000 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.682194 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.682195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.682195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.682320 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:39.682374 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.682492 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.682645 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260620 08:04:39.685144 7265 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:57234:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:04:39.685412 7265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:39.687562 7402 tablet_service.cc:1467] Processing CreateTablet for tablet fbb50fd014944f67baba98b9dc1e87ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efa777ad48324dd49de872bd1c9f1fb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:39.687575 7682 tablet_service.cc:1467] Processing CreateTablet for tablet fbb50fd014944f67baba98b9dc1e87ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efa777ad48324dd49de872bd1c9f1fb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:39.687564 7542 tablet_service.cc:1467] Processing CreateTablet for tablet fbb50fd014944f67baba98b9dc1e87ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efa777ad48324dd49de872bd1c9f1fb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:04:39.687757 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb50fd014944f67baba98b9dc1e87ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.687757 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb50fd014944f67baba98b9dc1e87ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.687757 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb50fd014944f67baba98b9dc1e87ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.689098 7777 tablet_bootstrap.cc:492] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: Bootstrap starting.
I20260620 08:04:39.689334 7778 tablet_bootstrap.cc:492] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: Bootstrap starting.
I20260620 08:04:39.689355 7782 tablet_bootstrap.cc:492] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: Bootstrap starting.
I20260620 08:04:39.689461 7777 tablet_bootstrap.cc:654] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.689795 7782 tablet_bootstrap.cc:654] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.689810 7778 tablet_bootstrap.cc:654] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.690091 7777 tablet_bootstrap.cc:492] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: No bootstrap required, opened a new log
I20260620 08:04:39.690140 7777 ts_tablet_manager.cc:1397] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:39.690268 7777 raft_consensus.cc:348] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.690338 7777 raft_consensus.cc:374] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.690363 7777 raft_consensus.cc:729] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.690410 7777 consensus_queue.cc:260] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.690454 7782 tablet_bootstrap.cc:492] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: No bootstrap required, opened a new log
I20260620 08:04:39.690502 7782 ts_tablet_manager.cc:1397] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:39.690519 7777 ts_tablet_manager.cc:1428] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:39.690629 7781 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.690649 7782 raft_consensus.cc:348] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.690704 7781 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.690718 7782 raft_consensus.cc:374] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.690738 7782 raft_consensus.cc:729] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb77008afb704765bced49ed988d305a, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.690809 7782 consensus_queue.cc:260] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.690907 7782 ts_tablet_manager.cc:1428] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:39.691064 7785 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.691169 7785 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.691479 7778 tablet_bootstrap.cc:492] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: No bootstrap required, opened a new log
I20260620 08:04:39.691536 7778 ts_tablet_manager.cc:1397] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:04:39.691690 7778 raft_consensus.cc:348] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.691799 7778 raft_consensus.cc:374] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.691820 7778 raft_consensus.cc:729] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.691864 7778 consensus_queue.cc:260] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.691965 7778 ts_tablet_manager.cc:1428] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:39.692118 7783 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.692232 7783 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.804298 7781 raft_consensus.cc:482] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:39.804445 7781 raft_consensus.cc:504] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.804755 7781 leader_election.cc:288] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:39.805039 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb50fd014944f67baba98b9dc1e87ac" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" is_pre_election: true
I20260620 08:04:39.805145 7562 raft_consensus.cc:2454] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 0.
I20260620 08:04:39.805063 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb50fd014944f67baba98b9dc1e87ac" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a" is_pre_election: true
I20260620 08:04:39.805193 7702 raft_consensus.cc:2454] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 0.
I20260620 08:04:39.805337 7355 leader_election.cc:302] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, ec37b82c16a1474caf037a1d98a5f9b5; no voters:
I20260620 08:04:39.805485 7781 raft_consensus.cc:2781] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:39.805574 7781 raft_consensus.cc:482] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:39.805644 7781 raft_consensus.cc:3037] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.806519 7781 raft_consensus.cc:504] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.806690 7781 leader_election.cc:288] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:39.806932 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb50fd014944f67baba98b9dc1e87ac" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169"
I20260620 08:04:39.806978 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb50fd014944f67baba98b9dc1e87ac" candidate_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a"
I20260620 08:04:39.807041 7562 raft_consensus.cc:3037] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.807050 7702 raft_consensus.cc:3037] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:39.807830 7702 raft_consensus.cc:2454] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 1.
I20260620 08:04:39.807852 7562 raft_consensus.cc:2454] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec37b82c16a1474caf037a1d98a5f9b5 in term 1.
I20260620 08:04:39.808048 7355 leader_election.cc:302] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, ec37b82c16a1474caf037a1d98a5f9b5; no voters:
I20260620 08:04:39.808218 7781 raft_consensus.cc:2781] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:39.808285 7781 raft_consensus.cc:686] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 LEADER]: Becoming Leader. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Running, Role: LEADER
I20260620 08:04:39.808355 7781 consensus_queue.cc:237] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.808656 7788 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: TxnStatusTablet state changed. Reason: New leader ec37b82c16a1474caf037a1d98a5f9b5. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.808789 7788 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:39.809032 7821 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:04:39.809018 7265 catalog_manager.cc:5495] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:39.809160 7821 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:04:39.841642 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.838961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.31.115.193:35453)
0620 08:04:39.839031 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.839033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.839044 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:39.839107 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.839108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.839120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.839183 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:39.839303 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.840048 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.840837 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.841150 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.841222 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.841367 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.841404 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.841457 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.841509 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.841530 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:39.841652 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.838876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.0.0.1:59318)
0620 08:04:39.838962 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.838975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.839031 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.839192 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.839193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.839194 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:39.839241 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.839244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.840161 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.840162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.840749 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.840752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.841229 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.841230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.841271 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.841288 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.841465 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.841471 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.841505 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.841546 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:04:39.842711 7702 raft_consensus.cc:1261] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Refusing update from remote peer ec37b82c16a1474caf037a1d98a5f9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:39.842765 7562 raft_consensus.cc:1261] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Refusing update from remote peer ec37b82c16a1474caf037a1d98a5f9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:39.842993 7788 consensus_queue.cc:1035] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:39.843166 7781 consensus_queue.cc:1035] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:39.843675 7785 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: TxnStatusTablet state changed. Reason: New leader ec37b82c16a1474caf037a1d98a5f9b5. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.843797 7785 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.844033 7783 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: TxnStatusTablet state changed. Reason: New leader ec37b82c16a1474caf037a1d98a5f9b5. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.844146 7783 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.844446 7781 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.844533 7781 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:39.844707 7791 mvcc.cc:205] Tried to move back new op lower bound from 7298837216633929728 to 7298837216495423488. Current Snapshot: MvccSnapshot[applied={T|T < 7298837216633929728 or (T in {7298837216633929728})}]
I20260620 08:04:39.845163 7783 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.845249 7783 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P 2c4ec33d7f9f41aead1dcbd65b1aa169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.846202 7788 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.846303 7788 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P ec37b82c16a1474caf037a1d98a5f9b5: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:04:39.846628 7785 tablet_replica.cc:439] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } } }
I20260620 08:04:39.846717 7785 tablet_replica.cc:442] T fbb50fd014944f67baba98b9dc1e87ac P bb77008afb704765bced49ed988d305a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:04:39.846776 7793 mvcc.cc:205] Tried to move back new op lower bound from 7298837216633929728 to 7298837216495423488. Current Snapshot: MvccSnapshot[applied={T|T < 7298837216633929728 or (T in {7298837216633929728})}]
I20260620 08:04:39.849082 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.846257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.31.115.195:45727)
0620 08:04:39.846302 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.846304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.846312 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:39.846349 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.846349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.846372 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.846437 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:39.846711 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.847396 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.848238 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.848454 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.848540 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.848719 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.848839 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.848932 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.848974 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.848997 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:04:39.855074 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.852081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:43323 (local address 127.31.115.193:35453)
0620 08:04:39.852154 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.852156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.852341 (+ 185us) server_negotiation.cc:407] Connection header received
0620 08:04:39.852396 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.852397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.852410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.852472 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:39.852595 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.853341 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.854325 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.854610 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.854684 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.854839 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.854875 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.854928 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.854972 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.854993 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:39.855093 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.852010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.31.115.195:43323)
0620 08:04:39.852253 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.852264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.852327 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.852479 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.852480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.852481 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:39.852535 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.852536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.853439 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.853440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.854239 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.854242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.854688 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.854689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.854740 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.854760 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.854934 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.854935 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.854969 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.855003 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260620 08:04:39.857802 7422 raft_consensus.cc:1261] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Refusing update from remote peer bb77008afb704765bced49ed988d305a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:39.858187 7562 raft_consensus.cc:1261] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Refusing update from remote peer bb77008afb704765bced49ed988d305a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:39.858174 7785 consensus_queue.cc:1035] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:39.858368 7785 consensus_queue.cc:1035] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:39.868238 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.865382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:43247 (local address 127.31.115.195:45727)
0620 08:04:39.865439 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.865441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.865449 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:39.865534 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.865535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.865547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.865602 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:39.865755 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.866242 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.867058 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.867428 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.867608 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.867940 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.867981 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.868051 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.868122 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.868146 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:04:39.868248 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.865191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:45727 (local address 127.31.115.193:43247)
0620 08:04:39.865359 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:39.865372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:39.865477 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:39.865632 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:39.865633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:39.865635 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:39.865687 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.865689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.866332 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.866333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.866949 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:39.866952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.867686 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.867687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:39.867772 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.867822 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:39.868065 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:39.868067 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:39.868117 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:04:39.868171 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260620 08:04:39.879854 7265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:39.879988 7265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:39.880697 7265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Sending DeleteTablet for 3 replicas of tablet 58a8b9090316497581a5dba0e00111fe
I20260620 08:04:39.881003 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a8b9090316497581a5dba0e00111fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:39 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59306
I20260620 08:04:39.881040 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a8b9090316497581a5dba0e00111fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:39 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50230
I20260620 08:04:39.881152 7800 tablet_replica.cc:330] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: stopping tablet replica
I20260620 08:04:39.881244 7800 raft_consensus.cc:2229] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:39.881306 7800 raft_consensus.cc:2258] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:39.881392 7802 tablet_replica.cc:330] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: stopping tablet replica
I20260620 08:04:39.881453 7802 raft_consensus.cc:2229] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:39.881505 7802 raft_consensus.cc:2258] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:39.881920 7802 ts_tablet_manager.cc:1905] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:39.883611 7802 ts_tablet_manager.cc:1918] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:39.883667 7802 log.cc:1196] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/58a8b9090316497581a5dba0e00111fe
I20260620 08:04:39.883890 7802 ts_tablet_manager.cc:1939] T 58a8b9090316497581a5dba0e00111fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting consensus metadata
I20260620 08:04:39.884630 7251 catalog_manager.cc:4841] TS 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935): tablet 58a8b9090316497581a5dba0e00111fe (table test-table [id=c7d9ba26a2c54992bb1fccf8573d6abb]) successfully deleted
I20260620 08:04:39.885253 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a8b9090316497581a5dba0e00111fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:39 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46278
I20260620 08:04:39.885366 7801 tablet_replica.cc:330] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: stopping tablet replica
I20260620 08:04:39.885418 7801 raft_consensus.cc:2229] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:39.885516 7801 raft_consensus.cc:2258] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:39.885929 7801 ts_tablet_manager.cc:1905] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:39.887777 7801 ts_tablet_manager.cc:1918] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:39.887837 7801 log.cc:1196] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/58a8b9090316497581a5dba0e00111fe
I20260620 08:04:39.888060 7801 ts_tablet_manager.cc:1939] T 58a8b9090316497581a5dba0e00111fe P bb77008afb704765bced49ed988d305a: Deleting consensus metadata
I20260620 08:04:39.888360 7800 ts_tablet_manager.cc:1905] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:39.890215 7800 ts_tablet_manager.cc:1918] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:39.890273 7800 log.cc:1196] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/58a8b9090316497581a5dba0e00111fe
I20260620 08:04:39.890501 7800 ts_tablet_manager.cc:1939] T 58a8b9090316497581a5dba0e00111fe P ec37b82c16a1474caf037a1d98a5f9b5: Deleting consensus metadata
I20260620 08:04:39.891333 7251 catalog_manager.cc:4841] TS bb77008afb704765bced49ed988d305a (127.31.115.195:45727): tablet 58a8b9090316497581a5dba0e00111fe (table test-table [id=c7d9ba26a2c54992bb1fccf8573d6abb]) successfully deleted
I20260620 08:04:39.891744 7251 catalog_manager.cc:4841] TS ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453): tablet 58a8b9090316497581a5dba0e00111fe (table test-table [id=c7d9ba26a2c54992bb1fccf8573d6abb]) successfully deleted
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:04:39.919811 7816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:39.911772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.31.115.254:41825)
0620 08:04:39.911897 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.911899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.911948 (+ 49us) server_negotiation.cc:407] Connection header received
0620 08:04:39.912051 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.912053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.912065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.912133 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.912949 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.913822 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.915084 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.915439 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.915522 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.919722 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57242: BlockingRecv error: recv got EOF from 127.0.0.1:57242 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 08:04:39 dist-test-slave-c4w8 krb5kdc[7220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM
I20260620 08:04:39.931275 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:39.921306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.31.115.254:41825)
0620 08:04:39.921381 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:39.921383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:39.921455 (+ 72us) server_negotiation.cc:407] Connection header received
0620 08:04:39.921532 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:39.921533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:39.921546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:39.921691 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:39.922381 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.923129 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.924317 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:39.924629 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:39.924706 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:39.927920 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:39.927923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:39.927923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:39.927934 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:39.929824 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.930254 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.930256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.930256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.930323 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:39.930551 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:39.930552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:39.930552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:39.930701 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:39.930808 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:39.931061 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 08:04:39.931153 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:04:39.933451 7265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:39.933677 7265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:39.935314 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 9cddd31d90444ac9a19e4be12c8fa722 (DEFAULT_TABLE table=test-table [id=e5ea4820f8e84fc78b5745f4c29c4cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.935400 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 9cddd31d90444ac9a19e4be12c8fa722 (DEFAULT_TABLE table=test-table [id=e5ea4820f8e84fc78b5745f4c29c4cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.935474 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cddd31d90444ac9a19e4be12c8fa722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.935534 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cddd31d90444ac9a19e4be12c8fa722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.937038 7777 tablet_bootstrap.cc:492] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Bootstrap starting.
I20260620 08:04:39.937029 7682 tablet_service.cc:1467] Processing CreateTablet for tablet 9cddd31d90444ac9a19e4be12c8fa722 (DEFAULT_TABLE table=test-table [id=e5ea4820f8e84fc78b5745f4c29c4cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:39.937155 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cddd31d90444ac9a19e4be12c8fa722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:39.937522 7777 tablet_bootstrap.cc:654] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.938512 7778 tablet_bootstrap.cc:492] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Bootstrap starting.
I20260620 08:04:39.938679 7777 tablet_bootstrap.cc:492] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: No bootstrap required, opened a new log
I20260620 08:04:39.938724 7777 ts_tablet_manager.cc:1397] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:04:39.938879 7778 tablet_bootstrap.cc:654] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.938864 7777 raft_consensus.cc:348] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.938931 7777 raft_consensus.cc:374] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.938953 7777 raft_consensus.cc:729] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.938997 7777 consensus_queue.cc:260] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.939082 7777 ts_tablet_manager.cc:1428] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:39.939213 7782 tablet_bootstrap.cc:492] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Bootstrap starting.
I20260620 08:04:39.939626 7782 tablet_bootstrap.cc:654] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:39.940426 7782 tablet_bootstrap.cc:492] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: No bootstrap required, opened a new log
I20260620 08:04:39.940479 7782 ts_tablet_manager.cc:1397] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:39.940625 7782 raft_consensus.cc:348] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.940692 7782 raft_consensus.cc:374] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.940711 7782 raft_consensus.cc:729] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb77008afb704765bced49ed988d305a, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.940752 7782 consensus_queue.cc:260] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.940833 7782 ts_tablet_manager.cc:1428] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:39.941851 7778 tablet_bootstrap.cc:492] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: No bootstrap required, opened a new log
I20260620 08:04:39.941910 7778 ts_tablet_manager.cc:1397] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 08:04:39.942054 7778 raft_consensus.cc:348] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.942128 7778 raft_consensus.cc:374] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:39.942147 7778 raft_consensus.cc:729] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Initialized, Role: FOLLOWER
I20260620 08:04:39.942202 7778 consensus_queue.cc:260] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:39.942289 7778 ts_tablet_manager.cc:1428] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:40.116088 7783 raft_consensus.cc:482] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:40.116217 7783 raft_consensus.cc:504] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.116645 7783 leader_election.cc:288] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:40.120046 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.117170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:50599 (local address 127.31.115.195:45727)
0620 08:04:40.117236 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:40.117238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:40.117247 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:40.117345 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:40.117346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:40.117355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:40.117411 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:40.117572 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.118156 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.119162 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.119701 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.119925 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.119937 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:40.119961 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 08:04:40.119991 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:04:40.120062 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.117080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:45727 (local address 127.31.115.194:50599)
0620 08:04:40.117171 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:40.117182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:40.117244 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:40.117435 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:40.117436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:40.117439 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:40.117500 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.117507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.118243 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.118244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.119067 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.119070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.119799 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.119800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.119864 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.119870 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:04:40.119919 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:04:40.119961 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":40,"threads_started":1}
I20260620 08:04:40.120558 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cddd31d90444ac9a19e4be12c8fa722" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a" is_pre_election: true
I20260620 08:04:40.120620 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.116946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:56663 (local address 127.31.115.193:35453)
0620 08:04:40.117051 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:40.117054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:40.117194 (+ 140us) server_negotiation.cc:407] Connection header received
0620 08:04:40.117372 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:40.117374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:40.117384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:40.117443 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:04:40.117713 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.118220 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.119622 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.120250 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.120314 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.120323 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:40.120516 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 08:04:40.120557 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 08:04:40.120630 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.116860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.31.115.194:56663)
0620 08:04:40.117066 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:40.117079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:40.117232 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:40.117474 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:40.117475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:40.117477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:04:40.117639 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.117642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.118312 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.118312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.119511 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.119514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.120360 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.120361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.120421 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.120424 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:40.120506 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 08:04:40.120543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260620 08:04:40.120669 7702 raft_consensus.cc:2454] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 0.
I20260620 08:04:40.120823 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cddd31d90444ac9a19e4be12c8fa722" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" is_pre_election: true
I20260620 08:04:40.120905 7422 raft_consensus.cc:2454] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 0.
I20260620 08:04:40.121097 7495 leader_election.cc:302] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, bb77008afb704765bced49ed988d305a; no voters:
I20260620 08:04:40.121284 7783 raft_consensus.cc:2781] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:40.121345 7783 raft_consensus.cc:482] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:40.121378 7783 raft_consensus.cc:3037] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:40.122193 7783 raft_consensus.cc:504] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.122344 7783 leader_election.cc:288] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 election: Requested vote from peers ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:40.122575 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cddd31d90444ac9a19e4be12c8fa722" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec37b82c16a1474caf037a1d98a5f9b5"
I20260620 08:04:40.122624 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cddd31d90444ac9a19e4be12c8fa722" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a"
I20260620 08:04:40.122673 7422 raft_consensus.cc:3037] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:40.122723 7702 raft_consensus.cc:3037] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:40.123344 7422 raft_consensus.cc:2454] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 1.
I20260620 08:04:40.123466 7702 raft_consensus.cc:2454] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 1.
I20260620 08:04:40.123600 7495 leader_election.cc:302] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, ec37b82c16a1474caf037a1d98a5f9b5; no voters:
I20260620 08:04:40.123827 7783 raft_consensus.cc:2781] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:40.123991 7783 raft_consensus.cc:686] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Running, Role: LEADER
I20260620 08:04:40.124081 7783 consensus_queue.cc:237] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.124861 7265 catalog_manager.cc:5495] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:40.174887 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.172132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.31.115.194:34935)
0620 08:04:40.172320 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:40.172322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:40.172329 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:40.172378 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:40.172379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:40.172399 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:40.172496 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:40.172684 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.173198 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.173986 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.174223 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.174302 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.174480 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:40.174616 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:40.174709 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:40.174750 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 08:04:40.174771 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260620 08:04:40.176056 7422 raft_consensus.cc:1261] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Refusing update from remote peer 2c4ec33d7f9f41aead1dcbd65b1aa169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:40.176100 7702 raft_consensus.cc:1261] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Refusing update from remote peer 2c4ec33d7f9f41aead1dcbd65b1aa169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:40.176430 7854 consensus_queue.cc:1035] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:40.176571 7783 consensus_queue.cc:1035] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:40.180361 7264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:40.180470 7264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:40.181108 7264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Sending DeleteTablet for 3 replicas of tablet 9cddd31d90444ac9a19e4be12c8fa722
I20260620 08:04:40.181430 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cddd31d90444ac9a19e4be12c8fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:40 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59306
I20260620 08:04:40.181528 7800 tablet_replica.cc:330] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: stopping tablet replica
I20260620 08:04:40.181517 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cddd31d90444ac9a19e4be12c8fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:40 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50230
I20260620 08:04:40.181581 7800 raft_consensus.cc:2229] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:40.181596 7802 tablet_replica.cc:330] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: stopping tablet replica
I20260620 08:04:40.181556 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cddd31d90444ac9a19e4be12c8fa722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:40 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46278
I20260620 08:04:40.181659 7800 raft_consensus.cc:2258] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:40.181661 7802 raft_consensus.cc:2229] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:40.181690 7801 tablet_replica.cc:330] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: stopping tablet replica
I20260620 08:04:40.181744 7801 raft_consensus.cc:2229] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:40.181747 7802 raft_consensus.cc:2258] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:40.181788 7801 raft_consensus.cc:2258] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:40.182214 7801 ts_tablet_manager.cc:1905] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:40.182163 7264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:04:40.182319 7264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:04:40.182328 7800 ts_tablet_manager.cc:1905] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:40.183943 7800 ts_tablet_manager.cc:1918] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:40.184006 7800 log.cc:1196] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/9cddd31d90444ac9a19e4be12c8fa722
I20260620 08:04:40.184068 7802 ts_tablet_manager.cc:1905] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:40.184262 7800 ts_tablet_manager.cc:1939] T 9cddd31d90444ac9a19e4be12c8fa722 P ec37b82c16a1474caf037a1d98a5f9b5: Deleting consensus metadata
I20260620 08:04:40.184621 7251 catalog_manager.cc:4841] TS ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453): tablet 9cddd31d90444ac9a19e4be12c8fa722 (table test-table [id=e5ea4820f8e84fc78b5745f4c29c4cc2]) successfully deleted
I20260620 08:04:40.185504 7802 ts_tablet_manager.cc:1918] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:40.185572 7802 log.cc:1196] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/9cddd31d90444ac9a19e4be12c8fa722
I20260620 08:04:40.185529 7402 tablet_service.cc:1467] Processing CreateTablet for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe (DEFAULT_TABLE table=test-table [id=3d796289cc194c5aa96a6a4cc159e2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:40.185694 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:40.185818 7802 ts_tablet_manager.cc:1939] T 9cddd31d90444ac9a19e4be12c8fa722 P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting consensus metadata
I20260620 08:04:40.186115 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe (DEFAULT_TABLE table=test-table [id=3d796289cc194c5aa96a6a4cc159e2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:40.186205 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:40.186815 7777 tablet_bootstrap.cc:492] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: Bootstrap starting.
I20260620 08:04:40.187178 7777 tablet_bootstrap.cc:654] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:40.187352 7778 tablet_bootstrap.cc:492] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Bootstrap starting.
I20260620 08:04:40.187693 7777 tablet_bootstrap.cc:492] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: No bootstrap required, opened a new log
I20260620 08:04:40.187757 7777 ts_tablet_manager.cc:1397] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:40.187767 7778 tablet_bootstrap.cc:654] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:40.187865 7777 raft_consensus.cc:348] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.187951 7777 raft_consensus.cc:374] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:40.187979 7777 raft_consensus.cc:729] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec37b82c16a1474caf037a1d98a5f9b5, State: Initialized, Role: FOLLOWER
I20260620 08:04:40.188022 7777 consensus_queue.cc:260] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.188129 7777 ts_tablet_manager.cc:1428] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:04:40.188225 7682 tablet_service.cc:1467] Processing CreateTablet for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe (DEFAULT_TABLE table=test-table [id=3d796289cc194c5aa96a6a4cc159e2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:04:40.188396 7801 ts_tablet_manager.cc:1918] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:04:40.188448 7801 log.cc:1196] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/9cddd31d90444ac9a19e4be12c8fa722
I20260620 08:04:40.188670 7801 ts_tablet_manager.cc:1939] T 9cddd31d90444ac9a19e4be12c8fa722 P bb77008afb704765bced49ed988d305a: Deleting consensus metadata
I20260620 08:04:40.188905 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:40.190076 7778 tablet_bootstrap.cc:492] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: No bootstrap required, opened a new log
I20260620 08:04:40.190171 7778 ts_tablet_manager.cc:1397] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 08:04:40.190176 7782 tablet_bootstrap.cc:492] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Bootstrap starting.
I20260620 08:04:40.190332 7778 raft_consensus.cc:348] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.190407 7778 raft_consensus.cc:374] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:40.190433 7778 raft_consensus.cc:729] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Initialized, Role: FOLLOWER
I20260620 08:04:40.190480 7782 tablet_bootstrap.cc:654] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:40.190481 7778 consensus_queue.cc:260] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.190588 7778 ts_tablet_manager.cc:1428] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:40.191043 7251 catalog_manager.cc:4841] TS 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935): tablet 9cddd31d90444ac9a19e4be12c8fa722 (table test-table [id=e5ea4820f8e84fc78b5745f4c29c4cc2]) successfully deleted
I20260620 08:04:40.191133 7782 tablet_bootstrap.cc:492] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: No bootstrap required, opened a new log
I20260620 08:04:40.191177 7782 ts_tablet_manager.cc:1397] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:40.191291 7782 raft_consensus.cc:348] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.191366 7782 raft_consensus.cc:374] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:40.191394 7782 raft_consensus.cc:729] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb77008afb704765bced49ed988d305a, State: Initialized, Role: FOLLOWER
I20260620 08:04:40.191437 7782 consensus_queue.cc:260] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.191526 7782 ts_tablet_manager.cc:1428] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:04:40.191542 7251 catalog_manager.cc:4841] TS bb77008afb704765bced49ed988d305a (127.31.115.195:45727): tablet 9cddd31d90444ac9a19e4be12c8fa722 (table test-table [id=e5ea4820f8e84fc78b5745f4c29c4cc2]) successfully deleted
I20260620 08:04:40.217018 7854 raft_consensus.cc:482] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:04:40.217190 7854 raft_consensus.cc:504] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.217362 7854 leader_election.cc:288] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:40.217633 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b744f4ae7d84e1aa1c7d81f1344f8fe" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a" is_pre_election: true
I20260620 08:04:40.217633 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b744f4ae7d84e1aa1c7d81f1344f8fe" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" is_pre_election: true
I20260620 08:04:40.217746 7702 raft_consensus.cc:2454] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 0.
I20260620 08:04:40.217775 7422 raft_consensus.cc:2454] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 0.
I20260620 08:04:40.217895 7495 leader_election.cc:302] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, bb77008afb704765bced49ed988d305a; no voters:
I20260620 08:04:40.218026 7854 raft_consensus.cc:2781] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:04:40.218079 7854 raft_consensus.cc:482] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:04:40.218101 7854 raft_consensus.cc:3037] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:40.219027 7854 raft_consensus.cc:504] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.219174 7854 leader_election.cc:288] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 election: Requested vote from peers ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453), bb77008afb704765bced49ed988d305a (127.31.115.195:45727)
I20260620 08:04:40.219444 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b744f4ae7d84e1aa1c7d81f1344f8fe" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec37b82c16a1474caf037a1d98a5f9b5"
I20260620 08:04:40.219434 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b744f4ae7d84e1aa1c7d81f1344f8fe" candidate_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb77008afb704765bced49ed988d305a"
I20260620 08:04:40.219522 7702 raft_consensus.cc:3037] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:40.219522 7422 raft_consensus.cc:3037] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:40.220422 7702 raft_consensus.cc:2454] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 1.
I20260620 08:04:40.220422 7422 raft_consensus.cc:2454] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4ec33d7f9f41aead1dcbd65b1aa169 in term 1.
I20260620 08:04:40.220677 7495 leader_election.cc:302] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4ec33d7f9f41aead1dcbd65b1aa169, ec37b82c16a1474caf037a1d98a5f9b5; no voters:
I20260620 08:04:40.220786 7854 raft_consensus.cc:2781] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:40.220846 7854 raft_consensus.cc:686] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4ec33d7f9f41aead1dcbd65b1aa169, State: Running, Role: LEADER
I20260620 08:04:40.220906 7854 consensus_queue.cc:237] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } }
I20260620 08:04:40.221693 7265 catalog_manager.cc:5495] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4ec33d7f9f41aead1dcbd65b1aa169" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 34935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20260620 08:04:40.234699 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.231796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.31.115.194:51975)
0620 08:04:40.231973 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:40.231986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:40.232051 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:40.232178 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:40.232185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:40.232187 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:40.232267 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.232269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.232957 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.232958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.233862 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.233865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.234218 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.234219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.234289 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.234310 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:40.234480 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:40.234482 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:40.234538 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:04:40.234583 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260620 08:04:40.234724 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.231861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:51975 (local address 127.31.115.193:35453)
0620 08:04:40.231932 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:40.231935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:40.232053 (+ 118us) server_negotiation.cc:407] Connection header received
0620 08:04:40.232111 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:40.232112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:40.232122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:40.232164 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:40.232366 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.232874 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.233941 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.234138 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.234195 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.234393 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:40.234421 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:40.234459 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:40.234539 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 08:04:40.234557 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:04:40.237521 7702 raft_consensus.cc:1261] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Refusing update from remote peer 2c4ec33d7f9f41aead1dcbd65b1aa169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:40.237564 7422 raft_consensus.cc:1261] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Refusing update from remote peer 2c4ec33d7f9f41aead1dcbd65b1aa169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:04:40.237846 7860 consensus_queue.cc:1035] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb77008afb704765bced49ed988d305a" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:40.238000 7860 consensus_queue.cc:1035] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec37b82c16a1474caf037a1d98a5f9b5" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:04:40.245683 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.243296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:34935 (local address 127.31.115.193:36109)
0620 08:04:40.243338 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:40.243349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:40.243396 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:40.243545 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:40.243547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:40.243548 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:04:40.243595 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.243596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.244255 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.244256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.244823 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:40.244825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.245202 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.245204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:40.245264 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.245270 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:04:40.245444 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:04:40.245446 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:04:40.245496 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:04:40.245557 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 08:04:40.246021 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.243380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:36109 (local address 127.31.115.194:34935)
0620 08:04:40.243418 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:40.243421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:40.243430 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:40.243452 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:40.243453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:40.243463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:40.243540 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:04:40.243669 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.244167 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.244913 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:40.245127 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:40.245204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:40.245338 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:04:40.245381 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:04:40.245441 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:40.245920 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 08:04:40.245946 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:04:40.252173 7264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:40.252297 7264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:04:40.253149 7264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b98b0a7ea8074e9fb05a22123a0717af: Sending DeleteTablet for 3 replicas of tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe
I20260620 08:04:40.253513 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:40 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50230
I20260620 08:04:40.253513 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:40 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59306
I20260620 08:04:40.253640 7802 tablet_replica.cc:330] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: stopping tablet replica
I20260620 08:04:40.253679 7800 tablet_replica.cc:330] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: stopping tablet replica
I20260620 08:04:40.253695 7802 raft_consensus.cc:2229] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:04:40.253731 7800 raft_consensus.cc:2229] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:40.253759 7802 raft_consensus.cc:2258] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:40.253765 7800 raft_consensus.cc:2258] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:40.253995 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:04:40 UTC) from {username='oryx', principal='oryx/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46278
I20260620 08:04:40.254071 7800 ts_tablet_manager.cc:1905] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P ec37b82c16a1474caf037a1d98a5f9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:40.254102 7801 tablet_replica.cc:330] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: stopping tablet replica
I20260620 08:04:40.254146 7801 raft_consensus.cc:2229] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:04:40.254189 7801 raft_consensus.cc:2258] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:04:40.254408 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 7337
I20260620 08:04:40.255124 7802 ts_tablet_manager.cc:1905] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:40.256798 7802 ts_tablet_manager.cc:1918] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:40.256891 7802 log.cc:1196] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/1b744f4ae7d84e1aa1c7d81f1344f8fe
I20260620 08:04:40.257117 7802 ts_tablet_manager.cc:1939] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P 2c4ec33d7f9f41aead1dcbd65b1aa169: Deleting consensus metadata
I20260620 08:04:40.257462 7251 catalog_manager.cc:4841] TS 2c4ec33d7f9f41aead1dcbd65b1aa169 (127.31.115.194:34935): tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe (table test-table [id=3d796289cc194c5aa96a6a4cc159e2d6]) successfully deleted
I20260620 08:04:40.257845 7801 ts_tablet_manager.cc:1905] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:04:40.259045 7801 ts_tablet_manager.cc:1918] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:04:40.259121 7801 log.cc:1196] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/1b744f4ae7d84e1aa1c7d81f1344f8fe
I20260620 08:04:40.259327 7801 ts_tablet_manager.cc:1939] T 1b744f4ae7d84e1aa1c7d81f1344f8fe P bb77008afb704765bced49ed988d305a: Deleting consensus metadata
I20260620 08:04:40.259675 7251 catalog_manager.cc:4841] TS bb77008afb704765bced49ed988d305a (127.31.115.195:45727): tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe (table test-table [id=3d796289cc194c5aa96a6a4cc159e2d6]) successfully deleted
W20260620 08:04:40.263921 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:35453 (error 108)
I20260620 08:04:40.264182 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 7477
I20260620 08:04:40.265473 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:40.265212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35453 (local address 127.0.0.1:59330)
0620 08:04:40.265382 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:40.265434 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
W20260620 08:04:40.265623 7251 catalog_manager.cc:4584] TS ec37b82c16a1474caf037a1d98a5f9b5 (127.31.115.193:35453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b744f4ae7d84e1aa1c7d81f1344f8fe: Network error: Client connection negotiation failed: client connection to 127.31.115.193:35453: connect: Connection refused (error 111)
I20260620 08:04:40.270448 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 7617
I20260620 08:04:40.276818 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 7233
2026-06-20T08:04:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3097 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:40 dist-test-slave-c4w8 krb5kdc[7875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:40 dist-test-slave-c4w8 krb5kdc[7875](info): set up 2 sockets
Jun 20 08:04:40 dist-test-slave-c4w8 krb5kdc[7875](info): commencing operation
krb5kdc: starting...
W20260620 08:04:41.358690 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:41 dist-test-slave-c4w8 krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:41.485221 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:39941
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:44741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:41.573536 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:41.573848 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:41.573872 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:41.575091 7888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:41.575133 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:41.575147 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:41.575160 7888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:41.575172 7888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:41.576686 7888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:44741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:39941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:41.576956 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:41.577225 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:41.582371 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:41.582432 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:41.582383 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:41.582510 7888 server_base.cc:1034] running on GCE node
I20260620 08:04:41.582801 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:41.583120 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:41.584265 7888 hybrid_clock.cc:648] HybridClock initialized: now 1781942681584254 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T08:04:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1314 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:04:41 dist-test-slave-c4w8 krb5kdc[7901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:04:41 dist-test-slave-c4w8 krb5kdc[7901](info): set up 2 sockets
Jun 20 08:04:41 dist-test-slave-c4w8 krb5kdc[7901](info): commencing operation
krb5kdc: starting...
W20260620 08:04:42.650749 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:04:42 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:04:42.782111 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:37799
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:42.869554 7914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:42.869860 7914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:42.869889 7914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:42.871114 7914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:42.871160 7914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:42.871176 7914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:42.871188 7914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:42.871201 7914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:42.872804 7914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:42.873093 7914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:42.873350 7914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:42.878758 7919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:42.878815 7920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:42.878811 7922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:42.879315 7914 server_base.cc:1034] running on GCE node
I20260620 08:04:42.879670 7914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:42.880117 7914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:42.881309 7914 hybrid_clock.cc:648] HybridClock initialized: now 1781942682881283 us; error 42 us; skew 500 ppm
Jun 20 08:04:42 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942682, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:42.884138 7914 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:42.884634 7914 webserver.cc:460] Webserver started at http://127.31.115.254:39513/ using document root <none> and password file <none>
I20260620 08:04:42.884898 7914 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:42.884946 7914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:42.885090 7914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:42.886389 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "449df2724a364b6d9c9302840768fce9"
format_stamp: "Formatted at 2026-06-20 08:04:42 on dist-test-slave-c4w8"
server_key: "539a39c87b0a5a162b8e88a1d1cd5c96"
server_key_iv: "1c6123147520bf41c3f7c808052f100a"
server_key_version: "encryptionkey@0"
I20260620 08:04:42.886793 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "449df2724a364b6d9c9302840768fce9"
format_stamp: "Formatted at 2026-06-20 08:04:42 on dist-test-slave-c4w8"
server_key: "539a39c87b0a5a162b8e88a1d1cd5c96"
server_key_iv: "1c6123147520bf41c3f7c808052f100a"
server_key_version: "encryptionkey@0"
I20260620 08:04:42.888314 7914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:42.889390 7929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:42.889707 7914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:42.889851 7914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "449df2724a364b6d9c9302840768fce9"
format_stamp: "Formatted at 2026-06-20 08:04:42 on dist-test-slave-c4w8"
server_key: "539a39c87b0a5a162b8e88a1d1cd5c96"
server_key_iv: "1c6123147520bf41c3f7c808052f100a"
server_key_version: "encryptionkey@0"
I20260620 08:04:42.889952 7914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:42.898464 7914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:42.900951 7914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:42.901139 7914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:42.905929 7914 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:37799
I20260620 08:04:42.905985 7981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:37799 every 8 connection(s)
I20260620 08:04:42.906378 7914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:42.907269 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 7914
I20260620 08:04:42.907398 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:42.907421 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:42.907615 32207 external_mini_cluster.cc:1428] Setting key 79b013e25120703c01a4a28bfbe776bc
I20260620 08:04:42.910995 7982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Bootstrap starting.
I20260620 08:04:42.911808 7982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:42.912127 7982 log.cc:824] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:42.913029 7982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: No bootstrap required, opened a new log
Jun 20 08:04:42 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:42.915927 7982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:42.916078 7982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:42.916100 7982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449df2724a364b6d9c9302840768fce9, State: Initialized, Role: FOLLOWER
I20260620 08:04:42.916250 7982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:42.916352 7982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:42.916399 7982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:42.916471 7982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:42.917114 7982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:42.917245 7982 leader_election.cc:302] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449df2724a364b6d9c9302840768fce9; no voters:
I20260620 08:04:42.917443 7982 leader_election.cc:288] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:42.917491 7987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:42.917719 7987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 LEADER]: Becoming Leader. State: Replica: 449df2724a364b6d9c9302840768fce9, State: Running, Role: LEADER
I20260620 08:04:42.917770 7982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:42.917862 7987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:42.918484 7989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449df2724a364b6d9c9302840768fce9. Latest consensus state: current_term: 1 leader_uuid: "449df2724a364b6d9c9302840768fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } } }
I20260620 08:04:42.918576 7989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:42.918927 7988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "449df2724a364b6d9c9302840768fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } } }
I20260620 08:04:42.919003 7988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:42.919407 7996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:42.920083 7996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:42.921017 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:42.908257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.31.115.254:37799)
0620 08:04:42.908475 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:42.908494 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:42.908520 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:04:42.909043 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:42.909050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:42.909103 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:42.909228 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:42.910128 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:42.911282 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:42.912449 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:42.913051 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:42.913203 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:42.916291 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:42.916295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:42.916295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:42.916308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:42.918576 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:42.919076 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:42.919079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:42.919080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:42.919288 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:42.919608 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:42.919609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:42.919610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:42.920022 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:42.920174 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:42.920654 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 08:04:42.920809 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":118,"threads_started":1}
I20260620 08:04:42.921710 7996 catalog_manager.cc:1347] Generated new cluster ID: f73faf00e3f54b7e8de519a61f20b417
I20260620 08:04:42.921761 7996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:42.939296 7996 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:42.939966 7996 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:42.943284 7996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Generated new TSK 0
I20260620 08:04:42.943511 7996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:04:42.973795 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37799
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:43.059975 8008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:43.060189 8008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:43.060215 8008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:43.061447 8008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:43.061498 8008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:43.061527 8008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:43.063077 8008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:43.063374 8008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:43.063637 8008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:43.068802 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.068838 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.068825 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:43.069273 8008 server_base.cc:1034] running on GCE node
I20260620 08:04:43.069543 8008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:43.069871 8008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:43.071101 8008 hybrid_clock.cc:648] HybridClock initialized: now 1781942683071095 us; error 40 us; skew 500 ppm
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:43.073920 8008 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:04:43.074420 8008 webserver.cc:460] Webserver started at http://127.31.115.193:32845/ using document root <none> and password file <none>
I20260620 08:04:43.074699 8008 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:43.074774 8008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:43.074934 8008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:43.076174 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "3f1f7b2923dd44509292057ea5d9e96b"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "91f5baec40918c85e77a07cfef3d6919"
server_key_iv: "e256593bcbe24e464e076070258ee17f"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.076587 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "3f1f7b2923dd44509292057ea5d9e96b"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "91f5baec40918c85e77a07cfef3d6919"
server_key_iv: "e256593bcbe24e464e076070258ee17f"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.078158 8008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:04:43.079074 8023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.079306 8008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.079437 8008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "3f1f7b2923dd44509292057ea5d9e96b"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "91f5baec40918c85e77a07cfef3d6919"
server_key_iv: "e256593bcbe24e464e076070258ee17f"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.079599 8008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:43.090667 8008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:43.093371 8008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:43.093556 8008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:43.093865 8008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:43.094224 8008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:43.094285 8008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.094326 8008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:43.094363 8008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.100958 8008 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:37597
I20260620 08:04:43.101003 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:37597 every 8 connection(s)
I20260620 08:04:43.101383 8008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:43.109197 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8008
I20260620 08:04:43.109308 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:43.109506 32207 external_mini_cluster.cc:1428] Setting key bbdf90c66abba6afcd502de5c5174333
I20260620 08:04:43.112329 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.102419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:59397 (local address 127.31.115.254:37799)
0620 08:04:43.102469 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.102471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.102732 (+ 261us) server_negotiation.cc:407] Connection header received
0620 08:04:43.103249 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.103250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.103262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.103330 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.104341 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.104943 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.106022 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.106287 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.106360 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.108634 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:43.108636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:43.108636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:43.108644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:43.110408 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:43.111018 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:43.111020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:43.111020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:43.111082 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:43.111401 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:43.111402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:43.111402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:43.111580 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:43.111656 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:43.112069 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 08:04:43.112192 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:04:43.112437 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.102383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:59397)
0620 08:04:43.102568 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.102618 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.103123 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.103393 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.103396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.103981 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.104229 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.104235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.105054 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.105055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.105918 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.105922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.106383 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.106384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.106499 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.107068 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.107078 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.108512 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:43.110515 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:43.110534 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:43.110536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:43.110911 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:43.111163 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:43.111164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:43.111165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:43.111323 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:43.111664 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:43.111670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:43.111854 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 08:04:43.111971 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 08:04:43.112195 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 08:04:43.112926 8137 heartbeater.cc:347] Connected to a master server at 127.31.115.254:37799
I20260620 08:04:43.113066 8137 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:43.113291 8137 heartbeater.cc:511] Master 127.31.115.254:37799 requested a full tablet report, sending...
I20260620 08:04:43.113993 7946 ts_manager.cc:194] Registered new tserver with Master: 3f1f7b2923dd44509292057ea5d9e96b (127.31.115.193:37597)
I20260620 08:04:43.114782 7946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:59397
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:04:43.136960 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37799
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:43.219849 8142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:43.220095 8142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:43.220161 8142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:43.221395 8142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:43.221455 8142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:43.221498 8142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:43.223030 8142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:43.223330 8142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:43.223577 8142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:43.228847 8147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.228899 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.228899 8148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:43.229252 8142 server_base.cc:1034] running on GCE node
I20260620 08:04:43.229460 8142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:43.229729 8142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:43.230890 8142 hybrid_clock.cc:648] HybridClock initialized: now 1781942683230870 us; error 36 us; skew 500 ppm
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:43.233575 8142 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:04:43.234139 8142 webserver.cc:460] Webserver started at http://127.31.115.194:33089/ using document root <none> and password file <none>
I20260620 08:04:43.234396 8142 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:43.234463 8142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:43.234615 8142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:43.235697 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "7c23e4cad6cf4744b19621265b6a5e33"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "f7b49044373b4291b102f1df2140437e"
server_key_iv: "561fd7cab44762470227e2637f75764e"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.236063 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "7c23e4cad6cf4744b19621265b6a5e33"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "f7b49044373b4291b102f1df2140437e"
server_key_iv: "561fd7cab44762470227e2637f75764e"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.237820 8142 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:04:43.238832 8157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.239076 8142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.239189 8142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "7c23e4cad6cf4744b19621265b6a5e33"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "f7b49044373b4291b102f1df2140437e"
server_key_iv: "561fd7cab44762470227e2637f75764e"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.239277 8142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:43.253286 8142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:43.255859 8142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:43.256006 8142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:43.256209 8142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:43.256477 8142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:43.256521 8142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.256547 8142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:43.256561 8142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.263370 8142 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:43607
I20260620 08:04:43.263449 8271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:43607 every 8 connection(s)
I20260620 08:04:43.263801 8142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:43.272419 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8142
I20260620 08:04:43.272548 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:43.272809 32207 external_mini_cluster.cc:1428] Setting key dd9eba6e1d1168bb9b28dbf50b6a6954
I20260620 08:04:43.274365 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.264599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:45477 (local address 127.31.115.254:37799)
0620 08:04:43.264690 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.264693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.264914 (+ 221us) server_negotiation.cc:407] Connection header received
0620 08:04:43.265448 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.265450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.265461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.265519 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.266602 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.267147 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.268175 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.268387 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.268456 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.270830 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:43.270831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:43.270832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:43.270839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:43.272341 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:43.273052 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:43.273054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:43.273054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:43.273150 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:43.273503 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:43.273505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:43.273505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:43.273685 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:43.273789 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:43.274209 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 08:04:43.274289 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:04:43.274669 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.264563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:45477)
0620 08:04:43.264761 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.264809 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.265311 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.265573 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.265576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.266244 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.266446 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.266452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.267249 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.267250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.268073 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.268076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.268472 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.268473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.268580 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.269107 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.269116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.270713 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:43.272442 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:43.272452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:43.272454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:43.272895 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:43.273250 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:43.273251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:43.273251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:43.273410 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:43.273833 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:43.273841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:43.274026 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 08:04:43.274169 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 08:04:43.274371 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260620 08:04:43.275867 8272 heartbeater.cc:347] Connected to a master server at 127.31.115.254:37799
I20260620 08:04:43.275987 8272 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:43.276204 8272 heartbeater.cc:511] Master 127.31.115.254:37799 requested a full tablet report, sending...
I20260620 08:04:43.276680 7946 ts_manager.cc:194] Registered new tserver with Master: 7c23e4cad6cf4744b19621265b6a5e33 (127.31.115.194:43607)
I20260620 08:04:43.277101 7946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:45477
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:04:43.301482 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37799
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:43.386478 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:43.386689 8277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:43.386714 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:43.387868 8277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:43.387907 8277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:43.387934 8277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:43.389351 8277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:43.389545 8277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:43.389797 8277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:43.394904 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.394929 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.394907 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:43.395184 8277 server_base.cc:1034] running on GCE node
I20260620 08:04:43.395444 8277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:43.395715 8277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:43.396903 8277 hybrid_clock.cc:648] HybridClock initialized: now 1781942683396875 us; error 51 us; skew 500 ppm
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:43.399575 8277 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:04:43.400107 8277 webserver.cc:460] Webserver started at http://127.31.115.195:40069/ using document root <none> and password file <none>
I20260620 08:04:43.400368 8277 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:43.400437 8277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:43.400584 8277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:43.401721 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "f3d114d3173d4b28b58d88a6fc6782bb"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "64a632624633316ba4fdb619f616122f"
server_key_iv: "57c8c8695cd389f9701a6d21ed4c72ad"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.402089 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "f3d114d3173d4b28b58d88a6fc6782bb"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "64a632624633316ba4fdb619f616122f"
server_key_iv: "57c8c8695cd389f9701a6d21ed4c72ad"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.403501 8277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:43.404374 8292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.404599 8277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.404703 8277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "f3d114d3173d4b28b58d88a6fc6782bb"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "64a632624633316ba4fdb619f616122f"
server_key_iv: "57c8c8695cd389f9701a6d21ed4c72ad"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.404762 8277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:43.420775 8277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:43.423247 8277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:43.423364 8277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:43.423568 8277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:43.423903 8277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:43.423940 8277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.423964 8277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:43.423976 8277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.430052 8277 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:42243
I20260620 08:04:43.430136 8405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:42243 every 8 connection(s)
I20260620 08:04:43.430460 8277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:04:43.437201 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8277
I20260620 08:04:43.437292 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:43.437480 32207 external_mini_cluster.cc:1428] Setting key 4e8c18486c191b418ed79c33dc3c3805
I20260620 08:04:43.440742 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.431206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:43013 (local address 127.31.115.254:37799)
0620 08:04:43.431279 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.431281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.431505 (+ 224us) server_negotiation.cc:407] Connection header received
0620 08:04:43.432063 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.432064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.432077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.432144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.433195 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.433797 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.434830 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.435057 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.435132 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.437671 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:43.437673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:43.437674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:43.437691 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:43.439384 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:43.439776 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:43.439777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:43.439777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:43.439835 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:43.440093 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:43.440094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:43.440095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:43.440239 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:43.440320 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:43.440606 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 08:04:43.440677 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:04:43.440878 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.431183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:43013)
0620 08:04:43.431376 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.431412 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.431932 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.432220 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.432224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.432851 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.433089 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.433095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.433890 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.433891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.434735 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.434738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.435158 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.435159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.435281 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.435869 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.435878 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.437528 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:43.439461 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:43.439468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:43.439471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:43.439707 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:43.439895 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:04:43.439896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:04:43.439897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:04:43.439998 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:04:43.440314 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:43.440319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:43.440440 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 08:04:43.440526 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:04:43.440654 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260620 08:04:43.441300 8406 heartbeater.cc:347] Connected to a master server at 127.31.115.254:37799
I20260620 08:04:43.441411 8406 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:43.441601 8406 heartbeater.cc:511] Master 127.31.115.254:37799 requested a full tablet report, sending...
I20260620 08:04:43.442060 7946 ts_manager.cc:194] Registered new tserver with Master: f3d114d3173d4b28b58d88a6fc6782bb (127.31.115.195:42243)
I20260620 08:04:43.442440 7946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:43013
I20260620 08:04:43.449762 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab.
Entry for principal oryx/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab.
WARNING: no policy specified for oryx/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab.
Entry for principal oryx/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab.
WARNING: no policy specified for oryx/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab.
Entry for principal oryx/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab.
I20260620 08:04:43.524302 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8008
I20260620 08:04:43.529307 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8142
I20260620 08:04:43.534518 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8277
I20260620 08:04:43.539252 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 7914
I20260620 08:04:43.544704 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:37799
--webserver_interface=127.31.115.254
--webserver_port=39513
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:04:43.630885 8415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:43.631106 8415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:43.631134 8415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:43.632380 8415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:43.632431 8415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:43.632455 8415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:43.632468 8415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:43.632481 8415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:43.634100 8415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=39513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:43.634384 8415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:43.634641 8415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:43.639856 8421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.639894 8423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.639851 8420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:43.640198 8415 server_base.cc:1034] running on GCE node
I20260620 08:04:43.640465 8415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:43.640769 8415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:43.641935 8415 hybrid_clock.cc:648] HybridClock initialized: now 1781942683641890 us; error 59 us; skew 500 ppm
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:43.644750 8415 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:04:43.645267 8415 webserver.cc:460] Webserver started at http://127.31.115.254:39513/ using document root <none> and password file <none>
I20260620 08:04:43.645579 8415 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:43.645736 8415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:43.647235 8415 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.647980 8430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.648169 8415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.648278 8415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "449df2724a364b6d9c9302840768fce9"
format_stamp: "Formatted at 2026-06-20 08:04:42 on dist-test-slave-c4w8"
server_key: "539a39c87b0a5a162b8e88a1d1cd5c96"
server_key_iv: "1c6123147520bf41c3f7c808052f100a"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.648662 8415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:43.658701 8415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:43.661152 8415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:43.661325 8415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:43.665947 8415 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:37799
I20260620 08:04:43.666011 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:37799 every 8 connection(s)
I20260620 08:04:43.666421 8415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:43.668017 8483 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:04:43.668749 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Bootstrap starting.
I20260620 08:04:43.669828 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8415
I20260620 08:04:43.670151 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:37597
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:37799
--webserver_port=32845
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:04:43.677989 8483 log.cc:824] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:43.680090 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:04:43.680414 8483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Bootstrap complete.
I20260620 08:04:43.682860 8483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:43.683105 8483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449df2724a364b6d9c9302840768fce9, State: Initialized, Role: FOLLOWER
I20260620 08:04:43.683318 8483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:43.683382 8483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:43.683449 8483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:43.683498 8483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:04:43.684564 8483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:43.684657 8483 leader_election.cc:302] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449df2724a364b6d9c9302840768fce9; no voters:
I20260620 08:04:43.684883 8483 leader_election.cc:288] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:04:43.684926 8487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:04:43.685088 8487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [term 2 LEADER]: Becoming Leader. State: Replica: 449df2724a364b6d9c9302840768fce9, State: Running, Role: LEADER
I20260620 08:04:43.685189 8483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:43.685308 8487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } }
I20260620 08:04:43.685680 8489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "449df2724a364b6d9c9302840768fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } } }
I20260620 08:04:43.685802 8489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:43.685678 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449df2724a364b6d9c9302840768fce9. Latest consensus state: current_term: 2 leader_uuid: "449df2724a364b6d9c9302840768fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449df2724a364b6d9c9302840768fce9" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 37799 } } }
I20260620 08:04:43.686149 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:43.686434 8496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:43.687083 8496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:43.687217 8496 catalog_manager.cc:1259] Loaded cluster ID: f73faf00e3f54b7e8de519a61f20b417
I20260620 08:04:43.687256 8496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:43.688405 8496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:43.688609 8496 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 449df2724a364b6d9c9302840768fce9: Loaded TSK: 0
I20260620 08:04:43.688834 8496 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:04:43.754781 8485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:43.755029 8485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:43.755072 8485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:43.756345 8485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:43.756426 8485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:43.756474 8485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:43.758574 8485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.193.keytab
--principal=oryx/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:37597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=32845
--tserver_master_addrs=127.31.115.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:43.758872 8485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:43.759133 8485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:43.765177 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.765213 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.765168 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:43.765527 8485 server_base.cc:1034] running on GCE node
I20260620 08:04:43.765765 8485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:43.766021 8485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:43.767204 8485 hybrid_clock.cc:648] HybridClock initialized: now 1781942683767190 us; error 31 us; skew 500 ppm
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:43.772715 8485 init.cc:377] Logged in from keytab as oryx/127.31.115.193@KRBTEST.COM (short username oryx)
I20260620 08:04:43.773258 8485 webserver.cc:460] Webserver started at http://127.31.115.193:32845/ using document root <none> and password file <none>
I20260620 08:04:43.773527 8485 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:43.773583 8485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:43.775202 8485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.775964 8519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.776240 8485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:43.776342 8485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "3f1f7b2923dd44509292057ea5d9e96b"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "91f5baec40918c85e77a07cfef3d6919"
server_key_iv: "e256593bcbe24e464e076070258ee17f"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.776810 8485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:43.791816 8485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:43.794492 8485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:43.794654 8485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:43.794950 8485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:43.795341 8485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:43.795395 8485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.795429 8485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:43.795454 8485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.802922 8485 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:37597
I20260620 08:04:43.802958 8632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:37597 every 8 connection(s)
I20260620 08:04:43.803411 8485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:43.805686 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8485
I20260620 08:04:43.806008 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:43607
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:37799
--webserver_port=33089
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:43.814534 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.804143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:57337)
0620 08:04:43.804367 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.804418 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.805071 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.805705 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.805709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.806467 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.806765 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.806772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.808144 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.808145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.809402 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.809406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.810290 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.810292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.810467 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.811128 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.811140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.814257 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
W20260620 08:04:43.814668 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:43.804260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:57337 (local address 127.31.115.254:37799)
0620 08:04:43.804450 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.804459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.804518 (+ 59us) server_negotiation.cc:407] Connection header received
0620 08:04:43.805293 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.805299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.805345 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.805540 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.806920 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.808028 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.809513 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.810172 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.810299 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.814457 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:57337: BlockingRecv error: recv got EOF from 127.31.115.193:57337 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
W20260620 08:04:43.815153 8633 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:37799 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.115.254:37799: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:43.821027 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.815409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:52859)
0620 08:04:43.815465 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.815477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.815518 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.815699 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.815700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.815876 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.815938 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.815940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.816863 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.816864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.817828 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.817832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.818291 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.818292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.818361 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.818841 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.818846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.820948 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:04:43.821164 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:43.815479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:52859 (local address 127.31.115.254:37799)
0620 08:04:43.815516 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.815518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.815526 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:43.815588 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.815589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.815601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.815665 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.816024 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.816771 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.817922 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.818209 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.818284 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.821078 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:52859: BlockingRecv error: recv got EOF from 127.31.115.193:52859 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:43.827307 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.821438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:36881)
0620 08:04:43.821658 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.821714 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.821780 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.821923 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.821924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.822108 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.822178 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.822180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.823092 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.823093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.824023 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.824026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.824537 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.824538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.824602 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.825157 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.825169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.827215 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
W20260620 08:04:43.827379 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:43.821655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:36881 (local address 127.31.115.254:37799)
0620 08:04:43.821696 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.821698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.821782 (+ 84us) server_negotiation.cc:407] Connection header received
0620 08:04:43.821833 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.821833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.821845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.821904 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.822255 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.823002 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.824116 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.824437 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.824512 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.827316 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:36881: BlockingRecv error: recv got EOF from 127.31.115.193:36881 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:04:43.827728 8633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:04:43.896714 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:43.896947 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:43.896987 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:43.898211 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:43.898278 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:43.898327 8637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:43.899799 8637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.194.keytab
--principal=oryx/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:43607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=33089
--tserver_master_addrs=127.31.115.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:43.900084 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:43.900326 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:43.905488 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.905488 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:43.905514 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:43.906373 8637 server_base.cc:1034] running on GCE node
I20260620 08:04:43.906637 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:43.906884 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:43.908038 8637 hybrid_clock.cc:648] HybridClock initialized: now 1781942683908026 us; error 30 us; skew 500 ppm
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942683, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:43.910881 8637 init.cc:377] Logged in from keytab as oryx/127.31.115.194@KRBTEST.COM (short username oryx)
I20260620 08:04:43.911386 8637 webserver.cc:460] Webserver started at http://127.31.115.194:33089/ using document root <none> and password file <none>
I20260620 08:04:43.911636 8637 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:43.911721 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:43.913318 8637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:04:43.914194 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.914425 8637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:04:43.914537 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "7c23e4cad6cf4744b19621265b6a5e33"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "f7b49044373b4291b102f1df2140437e"
server_key_iv: "561fd7cab44762470227e2637f75764e"
server_key_version: "encryptionkey@0"
I20260620 08:04:43.914963 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:43.927663 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:43.930166 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:43.930342 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:43.930611 8637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:43.930944 8637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:43.931001 8637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.931044 8637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:43.931069 8637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:43.937305 8637 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:43607
I20260620 08:04:43.937352 8766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:43607 every 8 connection(s)
I20260620 08:04:43.937780 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:43.942282 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8637
I20260620 08:04:43.942664 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:42243
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:37799
--webserver_port=40069
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:04:43.945758 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:43.938560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:38347 (local address 127.31.115.254:37799)
0620 08:04:43.938614 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.938616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.938816 (+ 200us) server_negotiation.cc:407] Connection header received
0620 08:04:43.939357 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.939359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.939375 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.939428 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.940442 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.940940 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.942058 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.942274 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.942359 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.945699 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:38347: BlockingRecv error: recv got EOF from 127.31.115.194:38347 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:04:43.945823 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.938536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:38347)
0620 08:04:43.938685 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.938727 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.939245 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.939512 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.939517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.940122 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.940326 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.940332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.941049 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.941050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.941964 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.941967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.942368 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.942369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.942498 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.943053 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.943062 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.945592 (+ 2530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
W20260620 08:04:43.946117 8767 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:37799 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.115.254:37799: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:43.951174 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.946406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:59537)
0620 08:04:43.946454 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.946464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.946512 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.946656 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.946657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.946796 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.946841 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.946843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.947476 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.947477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.947977 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.947979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.948339 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.948340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.948409 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.948861 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.948872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.951054 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 08:04:43.951193 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:43.946433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:59537 (local address 127.31.115.254:37799)
0620 08:04:43.946471 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.946473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.946497 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:04:43.946583 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.946584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.946594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.946653 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.946926 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.947392 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.948063 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.948264 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.948342 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.951140 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:59537: BlockingRecv error: recv got EOF from 127.31.115.194:59537 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:43 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:43.956424 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:43.951554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:56809)
0620 08:04:43.951600 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:43.951611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:43.951647 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:43.951837 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:43.951838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:43.951975 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:43.952004 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.952005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.953127 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.953128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.953693 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:43.953695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.954252 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.954253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:43.954293 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.954757 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:43.954761 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:43.956373 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 08:04:43.956576 8767 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:04:43.956629 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:43.951629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:56809 (local address 127.31.115.254:37799)
0620 08:04:43.951728 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:43.951730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:43.951737 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:04:43.951755 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:43.951756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:43.951764 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:43.951832 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:43.952116 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.953024 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.953773 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:43.954165 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:43.954248 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:43.956480 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:56809: BlockingRecv error: recv got EOF from 127.31.115.194:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 08:04:44.030706 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:44.030913 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:44.030936 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:44.032055 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:44.032089 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:44.032116 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:44.033491 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/krb5kdc/oryx_127.31.115.195.keytab
--principal=oryx/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:42243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=40069
--tserver_master_addrs=127.31.115.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:44.033761 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:44.033941 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:44.038993 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:44.038981 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:44.038981 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:44.039404 8770 server_base.cc:1034] running on GCE node
I20260620 08:04:44.039651 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:44.039889 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:44.041046 8770 hybrid_clock.cc:648] HybridClock initialized: now 1781942684041036 us; error 31 us; skew 500 ppm
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942684, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:04:44.043638 8770 init.cc:377] Logged in from keytab as oryx/127.31.115.195@KRBTEST.COM (short username oryx)
I20260620 08:04:44.044165 8770 webserver.cc:460] Webserver started at http://127.31.115.195:40069/ using document root <none> and password file <none>
I20260620 08:04:44.044423 8770 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:44.044503 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:44.046000 8770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:04:44.046641 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:44.046833 8770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:04:44.046902 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "f3d114d3173d4b28b58d88a6fc6782bb"
format_stamp: "Formatted at 2026-06-20 08:04:43 on dist-test-slave-c4w8"
server_key: "64a632624633316ba4fdb619f616122f"
server_key_iv: "57c8c8695cd389f9701a6d21ed4c72ad"
server_key_version: "encryptionkey@0"
I20260620 08:04:44.047247 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:44.067525 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:44.069661 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:44.069783 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:44.069983 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:44.070302 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:44.070338 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:44.070361 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:44.070412 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:44.076452 8770 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:42243
I20260620 08:04:44.076496 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:42243 every 8 connection(s)
I20260620 08:04:44.076848 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:44.078691 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8770
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:04:44.085712 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:44.077655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:60921 (local address 127.31.115.254:37799)
0620 08:04:44.077727 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:44.077730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:44.077952 (+ 222us) server_negotiation.cc:407] Connection header received
0620 08:04:44.078496 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:44.078499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:44.078512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:44.078570 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:44.079756 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.080415 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.082008 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.082271 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.082330 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.085621 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:60921: BlockingRecv error: recv got EOF from 127.31.115.195:60921 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":35}
I20260620 08:04:44.085736 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:44.077661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:60921)
0620 08:04:44.077832 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:44.077868 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:44.078374 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:44.078649 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:44.078652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:44.079391 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:44.079674 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.079679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.080521 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.080553 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.081937 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.081942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.082358 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.082359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.082516 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.082960 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:44.082971 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:44.085497 (+ 2526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
W20260620 08:04:44.085986 8899 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:37799 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.115.254:37799: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
I20260620 08:04:44.087983 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:44.079244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.31.115.254:37799)
0620 08:04:44.079386 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:44.079388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:44.079396 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:44.079419 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:44.079420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:44.079431 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:44.079502 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:44.080123 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.080762 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.081897 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.082190 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.082261 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.084224 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:44.084230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:44.084230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:04:44.084242 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:44.086474 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:44.086860 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:44.086864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:44.086864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:44.086982 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:04:44.087224 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:04:44.087225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:04:44.087225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:04:44.087514 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:44.087700 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:44.087808 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 08:04:44.087892 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:44.090878 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:44.086249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:50845)
0620 08:04:44.086323 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:44.086334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:44.086386 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:44.086538 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:44.086539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:44.086685 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:44.086745 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.086747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.087663 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.087664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.088434 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.088436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.088870 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.088871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.088936 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.089260 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:44.089264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:44.090831 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 08:04:44.091073 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:44.086302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:50845 (local address 127.31.115.254:37799)
0620 08:04:44.086341 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:44.086343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:44.086368 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:04:44.086443 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:44.086444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:44.086455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:44.086517 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:44.086837 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.087533 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.088538 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.088782 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.088869 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.091006 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:50845: BlockingRecv error: recv got EOF from 127.31.115.195:50845 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:44.094801 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:44.091188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:37809)
0620 08:04:44.091278 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:44.091288 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:44.091324 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:44.091440 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:44.091441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:44.091541 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:44.091567 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.091568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.092156 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.092157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.092617 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.092618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.092904 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.092905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.092940 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.093264 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:44.093268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:44.094755 (+ 1487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 08:04:44.094898 8636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:44.091266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:37809 (local address 127.31.115.254:37799)
0620 08:04:44.091324 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:44.091326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:44.091357 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:04:44.091376 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:44.091377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:44.091387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:44.091436 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:44.091651 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.092083 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.092684 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.092848 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.092895 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.094833 (+ 1938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:37809: BlockingRecv error: recv got EOF from 127.31.115.195:37809 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:04:44.094977 8899 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:44.833139 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:44.828151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:47191)
0620 08:04:44.828332 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:44.828345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:44.828402 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:44.828707 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:44.828708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:44.828871 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:44.828921 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.828923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.829637 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.829638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.830213 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.830215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.830600 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.830602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.830655 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.831141 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:44.831146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:44.833075 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
W20260620 08:04:44.833294 8905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:44.828267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:47191 (local address 127.31.115.254:37799)
0620 08:04:44.828398 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:44.828400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:44.828409 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:44.828452 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:44.828453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:44.828463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:44.828521 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:44.829006 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.829506 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.830305 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.830504 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.830612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.833195 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:47191: BlockingRecv error: recv got EOF from 127.31.115.193:47191 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:44 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:44.962292 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:44.957015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:40047)
0620 08:04:44.957169 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:44.957179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:44.957230 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:44.957417 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:44.957418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:44.957685 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:44.957762 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.957765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.958764 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.958766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.959398 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:44.959400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.959781 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.959782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:44.959826 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.960307 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:44.960312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:44.962225 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
W20260620 08:04:44.962402 8905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:44.957120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:40047 (local address 127.31.115.254:37799)
0620 08:04:44.957202 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:44.957204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:44.957222 (+ 18us) server_negotiation.cc:407] Connection header received
0620 08:04:44.957293 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:44.957294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:44.957304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:44.957379 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:44.957873 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.958678 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.959496 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:44.959702 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:44.959761 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:44.962332 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:40047: BlockingRecv error: recv got EOF from 127.31.115.194:40047 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 08:04:45 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:45 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:45.100541 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:45.095437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:35881)
0620 08:04:45.095616 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:45.095627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:45.095680 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:45.095837 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:45.095837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:45.095996 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:45.096049 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:45.096050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.096833 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.096834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:45.097391 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:45.097394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.097921 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.097923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:45.097977 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:45.098443 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:45.098447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:45.100473 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260620 08:04:45.100703 8905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:45.095534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:35881 (local address 127.31.115.254:37799)
0620 08:04:45.095657 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:45.095659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:45.095674 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:04:45.095744 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:45.095745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:45.095758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:45.095817 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:45.096146 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.096725 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.097553 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.097808 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.097880 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:45.100638 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:35881: BlockingRecv error: recv got EOF from 127.31.115.195:35881 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 08:04:45 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:45 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:45.838769 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:45.833724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:32937)
0620 08:04:45.833864 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:45.833876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:45.833934 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:45.834103 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:45.834104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:45.834264 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:45.834315 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:45.834316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.835106 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.835108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:45.835704 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:45.835707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.836145 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.836146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:45.836201 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:45.836717 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:45.836722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:45.838665 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260620 08:04:45.838835 8909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:45.833829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:32937 (local address 127.31.115.254:37799)
0620 08:04:45.833949 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:45.833951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:45.833978 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:04:45.833997 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:45.833998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:45.834009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:45.834067 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:45.834439 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.834997 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.835823 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.836042 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.836107 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:45.838778 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:32937: BlockingRecv error: recv got EOF from 127.31.115.193:32937 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
Jun 20 08:04:45 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:45 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:45.967934 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:45.962962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:35703)
0620 08:04:45.963139 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:45.963152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:45.963239 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:45.963465 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:45.963466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:45.963630 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:45.963689 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:45.963690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.964378 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.964378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:45.965048 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:45.965051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.965447 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.965448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:45.965500 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:45.966054 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:45.966059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:45.967871 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":122,"threads_started":1}
W20260620 08:04:45.968194 8909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:45.963082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:35703 (local address 127.31.115.254:37799)
0620 08:04:45.963146 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:45.963148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:45.963297 (+ 149us) server_negotiation.cc:407] Connection header received
0620 08:04:45.963349 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:45.963350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:45.963364 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:45.963415 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:45.963783 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.964299 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.965143 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:45.965342 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:45.965403 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:45.968076 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:35703: BlockingRecv error: recv got EOF from 127.31.115.194:35703 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 08:04:46 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:46 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:46.106931 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:46.101173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:49329)
0620 08:04:46.101333 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:46.101350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:46.101427 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:46.101694 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:46.101695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:46.101899 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:46.101949 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:46.101951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.102803 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.102804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:46.103665 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:46.103668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.104079 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.104080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:46.104151 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:46.104706 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:46.104712 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:46.106826 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
W20260620 08:04:46.106992 8909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:46.101383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:49329 (local address 127.31.115.254:37799)
0620 08:04:46.101464 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:46.101466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:46.101477 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:04:46.101510 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:46.101511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:46.101525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:46.101697 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:46.102044 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.102691 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.103753 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.103985 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.104077 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:46.106921 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:49329: BlockingRecv error: recv got EOF from 127.31.115.195:49329 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 08:04:46 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:46 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:46.844646 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:46.839344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:39319)
0620 08:04:46.839506 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:46.839522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:46.839574 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:46.839759 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:46.839759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:46.839917 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:46.839970 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:46.839971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.840642 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.840643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:46.841249 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:46.841252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.841741 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.841742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:46.841797 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:46.842316 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:46.842322 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:46.844526 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
W20260620 08:04:46.844689 8913 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:46.839475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:39319 (local address 127.31.115.254:37799)
0620 08:04:46.839619 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:46.839621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:46.839629 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:46.839653 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:46.839654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:46.839664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:46.839734 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:46.840033 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.840537 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.841360 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.841564 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.841728 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:46.844612 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:39319: BlockingRecv error: recv got EOF from 127.31.115.193:39319 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
Jun 20 08:04:46 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:46 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:46.973501 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:46.968566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:43013)
0620 08:04:46.968731 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:46.968748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:46.968829 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:46.969054 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:46.969056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:46.969288 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:46.969340 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:46.969342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.970106 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.970107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:46.970660 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:46.970662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.971034 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.971036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:46.971094 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:46.971589 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:46.971595 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:46.973390 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
W20260620 08:04:46.973541 8913 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:46.968659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:43013 (local address 127.31.115.254:37799)
0620 08:04:46.968709 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:46.968711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:46.968829 (+ 118us) server_negotiation.cc:407] Connection header received
0620 08:04:46.968962 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:46.968963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:46.968976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:46.969044 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:46.969426 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.970021 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.970750 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:46.970948 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:46.971024 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:46.973477 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:43013: BlockingRecv error: recv got EOF from 127.31.115.194:43013 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 08:04:47 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:47 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:04:47.112640 8913 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:47.107762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:33919 (local address 127.31.115.254:37799)
0620 08:04:47.107849 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:47.107851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:47.107859 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:04:47.107894 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:47.107895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:47.107908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:47.107973 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:47.108315 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.108827 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.109526 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.109778 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.109864 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:47.112581 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:33919: BlockingRecv error: recv got EOF from 127.31.115.195:33919 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:04:47.112639 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:47.107566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:33919)
0620 08:04:47.107706 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:47.107722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:47.107804 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:47.107983 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:47.107984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:47.108167 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:47.108214 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:47.108216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.108912 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.108913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:47.109437 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:47.109438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.109866 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.109867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:47.109934 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:47.110378 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:47.110383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:47.112481 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
Jun 20 08:04:47 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:47 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:47.851327 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:47.845348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:48991)
0620 08:04:47.845567 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:47.845590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:47.845723 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:47.846033 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:47.846035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:47.846219 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:47.846274 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:47.846276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.847169 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.847170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:47.847797 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:47.847800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.848226 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.848227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:47.848288 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:47.848858 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:47.848863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:47.851234 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":147,"threads_started":1}
W20260620 08:04:47.851419 8917 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:47.845424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:48991 (local address 127.31.115.254:37799)
0620 08:04:47.845668 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:47.845674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:47.845691 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:04:47.845826 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:47.845828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:47.845842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:47.845944 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:47.846385 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.846976 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.847904 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.848119 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.848226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:47.851346 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:48991: BlockingRecv error: recv got EOF from 127.31.115.193:48991 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
Jun 20 08:04:47 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:47 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:47.979384 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:47.974145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:55299)
0620 08:04:47.974340 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:47.974353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:47.974441 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:47.974599 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:47.974599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:47.974788 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:47.974839 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:47.974841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.975585 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.975586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:47.976186 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:47.976189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.976618 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.976619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:47.976711 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:47.977228 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:47.977239 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:47.979323 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
W20260620 08:04:47.979628 8917 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:47.974241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:55299 (local address 127.31.115.254:37799)
0620 08:04:47.974354 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:47.974357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:47.974416 (+ 59us) server_negotiation.cc:407] Connection header received
0620 08:04:47.974506 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:47.974507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:47.974519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:47.974565 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:47.974963 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.975477 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.976289 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:47.976541 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:47.976606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:47.979488 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:55299: BlockingRecv error: recv got EOF from 127.31.115.194:55299 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 08:04:48 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:48 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:48.120203 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:48.113331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:46961)
0620 08:04:48.113538 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:48.113547 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:48.113722 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:48.114020 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:48.114021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:48.114217 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:48.114279 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:48.114281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.115088 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.115089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:48.115678 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:48.115680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.116124 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.116125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:48.116179 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:48.116726 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:48.116731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:48.120106 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
W20260620 08:04:48.120362 8917 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:48.113441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:46961 (local address 127.31.115.254:37799)
0620 08:04:48.113507 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:48.113509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:48.113685 (+ 176us) server_negotiation.cc:407] Connection header received
0620 08:04:48.113857 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:48.113858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:48.113873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:48.113942 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:48.114381 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.114995 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.115829 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.116021 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.116080 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:48.120260 (+ 4180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:46961: BlockingRecv error: recv got EOF from 127.31.115.195:46961 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 08:04:48 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:48 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:48.862432 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:48.851950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:60605)
0620 08:04:48.852080 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:48.852091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:48.852194 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:48.852384 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:48.852386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:48.852563 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:48.852619 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:48.852621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.853399 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.853401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:48.854096 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:48.854099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.854502 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.854504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:48.854564 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:48.859991 (+ 5427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:48.859998 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:48.862352 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260620 08:04:48.862670 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:48.851998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:60605 (local address 127.31.115.254:37799)
0620 08:04:48.852171 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:48.852173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:48.852186 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:04:48.852254 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:48.852255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:48.852267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:48.852338 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:48.852749 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.853300 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.854197 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.854404 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.854472 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:48.862576 (+ 8104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:60605: BlockingRecv error: recv got EOF from 127.31.115.193:60605 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
Jun 20 08:04:48 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:48 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:48.985306 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:48.979993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:39967)
0620 08:04:48.980168 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:48.980183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:48.980325 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:48.980516 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:48.980517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:48.980743 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:48.980794 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:48.980796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.981528 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.981529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:48.982215 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:48.982217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.982734 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.982735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:48.982796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:48.983317 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:48.983329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:48.985227 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
W20260620 08:04:48.985416 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:48.980260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:39967 (local address 127.31.115.254:37799)
0620 08:04:48.980349 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:48.980356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:48.980365 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:48.980395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:48.980396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:48.980407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:48.980464 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:48.980933 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.981446 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.982389 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:48.982648 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:48.982736 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:48.985337 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:39967: BlockingRecv error: recv got EOF from 127.31.115.194:39967 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 08:04:49 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:49 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:49.125998 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:49.121004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:59589)
0620 08:04:49.121170 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:49.121180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:49.121235 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:49.121441 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:49.121442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:49.121630 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:49.121681 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:49.121683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.122388 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.122389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:49.122956 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:49.122958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.123351 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.123352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:49.123398 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:49.123857 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:49.123862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:49.125913 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
W20260620 08:04:49.126169 8921 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:49.121143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:59589 (local address 127.31.115.254:37799)
0620 08:04:49.121216 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:49.121218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:49.121228 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:49.121310 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:49.121311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:49.121324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:49.121393 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:49.121802 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.122299 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.123049 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.123269 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.123330 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:49.126063 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:59589: BlockingRecv error: recv got EOF from 127.31.115.195:59589 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 08:04:49 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:49 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:49.868544 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:49.863058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:46563)
0620 08:04:49.863250 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:49.863268 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:49.863330 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:49.863539 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:49.863540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:49.863733 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:49.863794 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:49.863796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.864510 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.864511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:49.865068 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:49.865070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.865515 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.865516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:49.865572 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:49.866271 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:49.866278 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:49.868481 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
W20260620 08:04:49.868755 8925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:49.863220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:46563 (local address 127.31.115.254:37799)
0620 08:04:49.863367 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:49.863369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:49.863378 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:49.863404 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:49.863404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:49.863415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:49.863480 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:49.863897 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.864421 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.865167 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.865377 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.865444 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:49.868634 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:46563: BlockingRecv error: recv got EOF from 127.31.115.193:46563 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
Jun 20 08:04:49 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:49 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:49.992118 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:49.985992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:43675)
0620 08:04:49.986179 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:49.986191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:49.986261 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:49.986430 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:49.986431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:49.986644 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:49.986708 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:49.986711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.987589 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.987591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:49.988442 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:49.988446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.988958 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.988959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:49.989025 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:49.989509 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:49.989514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:49.992037 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
W20260620 08:04:49.992280 8925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:49.986037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:43675 (local address 127.31.115.254:37799)
0620 08:04:49.986115 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:49.986118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:49.986256 (+ 138us) server_negotiation.cc:407] Connection header received
0620 08:04:49.986319 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:49.986320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:49.986333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:49.986388 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:49.986950 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.987486 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.988627 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:49.988860 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:49.988927 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:49.992177 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:43675: BlockingRecv error: recv got EOF from 127.31.115.194:43675 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 08:04:50 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:50 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:50.131690 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:50.126525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:59871)
0620 08:04:50.126742 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:50.126755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:50.126847 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:50.127012 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:50.127013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:50.127198 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:50.127261 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:50.127263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.127930 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.127931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:50.128454 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:50.128456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.128839 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.128840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:50.128918 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:50.129399 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:50.129403 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:50.131583 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
W20260620 08:04:50.131822 8925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:50.126660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:59871 (local address 127.31.115.254:37799)
0620 08:04:50.126746 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:50.126748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:50.126826 (+ 78us) server_negotiation.cc:407] Connection header received
0620 08:04:50.126913 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:50.126914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:50.126928 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:50.127002 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:50.127350 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.127847 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.128540 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.128748 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.128836 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:50.131749 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:59871: BlockingRecv error: recv got EOF from 127.31.115.195:59871 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 08:04:50 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:50 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:50.874537 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:50.869258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:45003)
0620 08:04:50.869416 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:50.869434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:50.869488 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:50.869759 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:50.869760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:50.869930 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:50.869981 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:50.869983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.870711 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.870712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:50.871255 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:50.871257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.871704 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.871705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:50.871773 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:50.872293 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:50.872299 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:50.874419 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260620 08:04:50.874576 8929 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:50.869362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:45003 (local address 127.31.115.254:37799)
0620 08:04:50.869529 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:50.869531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:50.869541 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:04:50.869566 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:50.869567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:50.869578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:50.869709 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:50.870076 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.870587 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.871363 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.871566 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.871673 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:50.874500 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:45003: BlockingRecv error: recv got EOF from 127.31.115.193:45003 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
Jun 20 08:04:50 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:50 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:50.998354 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:50.992911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:36489)
0620 08:04:50.993127 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:50.993142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:50.993242 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:50.993463 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:50.993464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:50.993736 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:50.993801 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:50.993803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.994628 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.994629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:50.995244 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:50.995246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.995733 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.995734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:50.995787 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:50.996335 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:50.996341 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:50.998250 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
W20260620 08:04:50.998521 8929 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:50.993005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:36489 (local address 127.31.115.254:37799)
0620 08:04:50.993062 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:50.993066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:50.993236 (+ 170us) server_negotiation.cc:407] Connection header received
0620 08:04:50.993351 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:50.993352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:50.993367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:50.993451 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:50.993923 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.994509 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.995419 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:50.995628 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:50.995692 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:50.998422 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:36489: BlockingRecv error: recv got EOF from 127.31.115.194:36489 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 08:04:51 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:51 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:51.137964 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:51.132340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:57107)
0620 08:04:51.132659 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:51.132670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:51.132775 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:51.132934 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:51.132935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:51.133143 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:51.133196 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:51.133198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.134034 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.134035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:51.134635 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:51.134637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.135053 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.135054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:51.135119 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:51.135722 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:51.135727 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:51.137864 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":93,"threads_started":1}
W20260620 08:04:51.138051 8929 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:51.132425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:57107 (local address 127.31.115.254:37799)
0620 08:04:51.132533 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:51.132535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:51.132748 (+ 213us) server_negotiation.cc:407] Connection header received
0620 08:04:51.132838 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:51.132839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:51.132856 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:51.132917 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:51.133292 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.133935 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.134739 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.134957 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.135051 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:51.137961 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:57107: BlockingRecv error: recv got EOF from 127.31.115.195:57107 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 08:04:51 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:51 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:51.880344 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:51.875103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:56543)
0620 08:04:51.875255 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:51.875269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:51.875335 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:51.875526 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:51.875527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:51.875691 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:51.875743 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:51.875745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.876435 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.876437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:51.877050 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:51.877054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.877492 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.877493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:51.877547 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:51.878181 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:51.878189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:51.880275 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
W20260620 08:04:51.880563 8933 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:51.875221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:56543 (local address 127.31.115.254:37799)
0620 08:04:51.875370 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:51.875372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:51.875402 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:04:51.875423 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:51.875423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:51.875433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:51.875481 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:51.875819 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.876327 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.877165 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:51.877370 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:51.877437 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:51.880406 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:56543: BlockingRecv error: recv got EOF from 127.31.115.193:56543 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
Jun 20 08:04:51 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:51 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:52.004796 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:51.999098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:53971)
0620 08:04:51.999349 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:51.999360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:51.999426 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:51.999562 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:51.999563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:51.999759 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:51.999806 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:51.999808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.000715 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.000716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:52.001300 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:52.001302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.001739 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.001740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:52.001791 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:52.002594 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:52.002600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:52.004716 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
W20260620 08:04:52.004989 8933 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:51.999198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:53971 (local address 127.31.115.254:37799)
0620 08:04:51.999274 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:51.999276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:51.999429 (+ 153us) server_negotiation.cc:407] Connection header received
0620 08:04:51.999472 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:51.999472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:51.999485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:51.999539 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:51.999892 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.000531 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.001409 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.001659 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.001727 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:52.004883 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:53971: BlockingRecv error: recv got EOF from 127.31.115.194:53971 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:04:52 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:52 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:52.144170 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:52.138734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:54247)
0620 08:04:52.139035 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:52.139044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:52.139115 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:52.139305 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:52.139306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:52.139468 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:52.139567 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:52.139569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.140341 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.140342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:52.140866 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:52.140868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.141265 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.141266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:52.141314 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:52.141883 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:52.141888 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:52.144085 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":185,"threads_started":1}
W20260620 08:04:52.144412 8933 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:52.138856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:54247 (local address 127.31.115.254:37799)
0620 08:04:52.138965 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:52.138968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:52.139147 (+ 179us) server_negotiation.cc:407] Connection header received
0620 08:04:52.139180 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:52.139181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:52.139194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:52.139253 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:52.139667 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.140214 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.140959 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.141182 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.141253 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:52.144217 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:54247: BlockingRecv error: recv got EOF from 127.31.115.195:54247 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 08:04:52 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:52 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:52.886487 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:52.880887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:37623)
0620 08:04:52.881032 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:52.881045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:52.881107 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:52.881376 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:52.881377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:52.881717 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:52.881775 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:52.881777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.882547 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.882548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:52.883122 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:52.883124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.883531 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.883532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:52.883600 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:52.884122 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:52.884127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:52.886360 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
W20260620 08:04:52.886540 8937 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:52.880930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:37623 (local address 127.31.115.254:37799)
0620 08:04:52.881048 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:52.881050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:52.881088 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:04:52.881236 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:52.881236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:52.881251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:52.881321 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:52.881877 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.882459 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.883220 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:52.883458 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:52.883521 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:52.886458 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:37623: BlockingRecv error: recv got EOF from 127.31.115.193:37623 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
Jun 20 08:04:53 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:53 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:53.010524 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:53.005449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:55097)
0620 08:04:53.005690 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:53.005712 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:53.005793 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:53.005937 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:53.005938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:53.006123 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:53.006171 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:53.006173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.006865 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.006866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:53.007388 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:53.007390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.007782 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.007783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:53.007841 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:53.008341 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:53.008346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:53.010447 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":136,"threads_started":1}
W20260620 08:04:53.010833 8937 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:53.005543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:55097 (local address 127.31.115.254:37799)
0620 08:04:53.005657 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:53.005659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:53.005775 (+ 116us) server_negotiation.cc:407] Connection header received
0620 08:04:53.005855 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:53.005856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:53.005870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:53.005917 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:53.006258 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.006774 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.007477 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.007688 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.007787 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:53.010580 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:55097: BlockingRecv error: recv got EOF from 127.31.115.194:55097 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 08:04:53 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:53 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:53.150341 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:53.144928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:53647)
0620 08:04:53.145097 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:53.145109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:53.145177 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:53.145356 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:53.145357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:53.145547 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:53.145599 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:53.145635 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.146376 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.146378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:53.147025 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:53.147028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.147405 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.147407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:53.147522 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:53.147990 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:53.147995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:53.150252 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
W20260620 08:04:53.150439 8937 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:53.145060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:53647 (local address 127.31.115.254:37799)
0620 08:04:53.145144 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:53.145146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:53.145160 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:04:53.145252 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:53.145253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:53.145270 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:53.145322 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:53.145750 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.146261 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.147122 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.147305 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.147361 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:53.150371 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:53647: BlockingRecv error: recv got EOF from 127.31.115.195:53647 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 08:04:53 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:53 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:53.899359 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:53.887090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:42459)
0620 08:04:53.887479 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:53.887494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:53.887623 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:53.887794 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:53.887795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:53.888012 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:53.888077 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:53.888079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.888766 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.888767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:53.889312 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:53.889314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.889797 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.889799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:53.889863 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:53.897000 (+ 7137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:53.897007 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:53.899268 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":70,"threads_started":1}
W20260620 08:04:53.899459 8941 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:53.887171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:42459 (local address 127.31.115.254:37799)
0620 08:04:53.887311 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:53.887313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:53.887563 (+ 250us) server_negotiation.cc:407] Connection header received
0620 08:04:53.887714 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:53.887715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:53.887729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:53.887791 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:53.888158 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.888678 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.889414 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:53.889694 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:53.889793 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:53.899383 (+ 9590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:42459: BlockingRecv error: recv got EOF from 127.31.115.193:42459 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
Jun 20 08:04:54 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:54 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:54.016702 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:54.011214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:39555)
0620 08:04:54.011415 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:54.011426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:54.011499 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:54.011651 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:54.011651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:54.011821 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:54.011872 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:54.011873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.012580 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.012581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:54.013460 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:54.013462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.014035 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.014036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:54.014079 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:54.014555 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:54.014560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:54.016637 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
W20260620 08:04:54.016891 8941 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:54.011345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:39555 (local address 127.31.115.254:37799)
0620 08:04:54.011417 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:54.011419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:54.011496 (+ 77us) server_negotiation.cc:407] Connection header received
0620 08:04:54.011552 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:54.011552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:54.011564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:54.011615 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:54.011977 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.012493 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.013700 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.013961 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.014019 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:54.016790 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:39555: BlockingRecv error: recv got EOF from 127.31.115.194:39555 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 08:04:54 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:54 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:54.156111 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:54.150965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:48291)
0620 08:04:54.151134 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:54.151147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:54.151250 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:54.151382 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:54.151383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:54.151605 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:54.151654 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:54.151656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.152382 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.152383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:54.152915 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:54.152918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.153345 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.153346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:54.153394 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:54.153895 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:54.153900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:54.156044 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
W20260620 08:04:54.156276 8941 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:54.151114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:48291 (local address 127.31.115.254:37799)
0620 08:04:54.151207 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:54.151210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:54.151226 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:04:54.151309 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:54.151309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:54.151324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:54.151376 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:54.151745 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.152292 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.153023 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.153259 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.153338 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:54.156174 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:48291: BlockingRecv error: recv got EOF from 127.31.115.195:48291 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 08:04:54 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:54 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:54.905318 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:54.900017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:49033)
0620 08:04:54.900193 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:54.900203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:54.900258 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:54.900470 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:54.900473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:54.900654 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:54.900704 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:54.900705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.901470 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.901471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:54.902319 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:54.902322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.902759 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.902760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:54.902819 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:54.903360 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:54.903364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:54.905236 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
W20260620 08:04:54.905417 8945 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:54.900154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:49033 (local address 127.31.115.254:37799)
0620 08:04:54.900290 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:54.900292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:54.900301 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:54.900327 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:54.900328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:54.900338 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:54.900411 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:54.900831 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.901370 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.902433 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:54.902664 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:54.902759 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:54.905340 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:49033: BlockingRecv error: recv got EOF from 127.31.115.193:49033 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
Jun 20 08:04:55 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:55 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:55.023029 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:55.017422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:34351)
0620 08:04:55.017659 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:55.017674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:55.017757 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:55.017937 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:55.017938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:55.018137 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:55.018185 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:55.018188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.018947 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.018948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:55.019483 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:55.019485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.019879 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.019880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:55.019942 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:55.020729 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:55.020735 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:55.022926 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260620 08:04:55.023281 8945 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:55.017581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:34351 (local address 127.31.115.254:37799)
0620 08:04:55.017685 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:55.017687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:55.017749 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:04:55.017838 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:55.017840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:55.017855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:55.017930 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:55.018277 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.018859 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.019586 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.019786 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.019879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:55.023220 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:34351: BlockingRecv error: recv got EOF from 127.31.115.194:34351 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 08:04:55 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:55 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:04:55.161636 8945 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:55.156806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:43121 (local address 127.31.115.254:37799)
0620 08:04:55.156879 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:55.156882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:55.156982 (+ 100us) server_negotiation.cc:407] Connection header received
0620 08:04:55.157072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:55.157074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:55.157114 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:55.157195 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:55.157528 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.158092 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.158874 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.159093 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.159181 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:55.161548 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:43121: BlockingRecv error: recv got EOF from 127.31.115.195:43121 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:04:55.161652 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:55.156707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:43121)
0620 08:04:55.156884 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:55.156897 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:55.156987 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:55.157207 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:55.157208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:55.157394 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:55.157440 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:55.157442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.158223 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.158224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:55.158783 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:55.158785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.159183 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.159184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:55.159244 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:55.159730 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:55.159735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:55.161461 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
Jun 20 08:04:55 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:55 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:55.911296 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:55.905917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:49243)
0620 08:04:55.906132 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:55.906153 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:55.906228 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:55.906416 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:55.906418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:55.906615 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:55.906682 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:55.906684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.907405 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.907406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:55.907985 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:55.907988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.908373 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.908374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:55.908432 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:55.909035 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:55.909045 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:55.911221 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
W20260620 08:04:55.911433 8949 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:55.905993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:49243 (local address 127.31.115.254:37799)
0620 08:04:55.906164 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:55.906166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:55.906207 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:04:55.906287 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:55.906288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:55.906301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:55.906370 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:55.906779 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.907303 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.908083 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:55.908293 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:55.908352 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:55.911336 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:49243: BlockingRecv error: recv got EOF from 127.31.115.193:49243 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
Jun 20 08:04:56 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:56 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:56.029382 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:56.023616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:32909)
0620 08:04:56.023858 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:56.023876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:56.023978 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:56.024293 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:56.024294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:56.024497 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:56.024557 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:56.024559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.025418 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.025419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:56.026115 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:56.026118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.026539 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.026540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:56.026606 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:56.027192 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:56.027198 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:56.029321 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
W20260620 08:04:56.029594 8949 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:56.023728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:32909 (local address 127.31.115.254:37799)
0620 08:04:56.023821 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:56.023830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:56.023953 (+ 123us) server_negotiation.cc:407] Connection header received
0620 08:04:56.024162 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:56.024164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:56.024187 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:56.024264 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:56.024729 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.025309 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.026214 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.026438 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.026539 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:56.029514 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:32909: BlockingRecv error: recv got EOF from 127.31.115.194:32909 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 08:04:56 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:56 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:56.167484 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:56.162184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:55843)
0620 08:04:56.162369 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:56.162386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:56.162458 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:56.162696 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:56.162697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:56.162886 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:56.162934 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:56.162936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.163643 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.163645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:56.164222 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:56.164224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.164612 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.164613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:56.164678 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:56.165184 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:56.165189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:56.167360 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
W20260620 08:04:56.167685 8949 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:56.162329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:55843 (local address 127.31.115.254:37799)
0620 08:04:56.162402 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:56.162404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:56.162437 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:04:56.162592 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:56.162593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:56.162607 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:56.162684 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:56.163016 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.163550 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.164316 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.164515 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.164609 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:56.167467 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:55843: BlockingRecv error: recv got EOF from 127.31.115.195:55843 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:04:56 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:56 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:56.917018 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:56.911883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:34711)
0620 08:04:56.912074 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:56.912087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:56.912155 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:56.912392 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:56.912394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:56.912607 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:56.912659 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:56.912660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.913478 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.913479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:56.914054 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:56.914056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.914421 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.914423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:56.914465 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:56.914977 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:56.914982 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:56.916954 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
W20260620 08:04:56.917232 8953 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:56.911995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:34711 (local address 127.31.115.254:37799)
0620 08:04:56.912110 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:56.912112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:56.912137 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:04:56.912253 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:56.912254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:56.912268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:56.912335 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:56.912762 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.913381 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.914141 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:56.914345 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:56.914409 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:56.917090 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:34711: BlockingRecv error: recv got EOF from 127.31.115.193:34711 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
Jun 20 08:04:57 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:57 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:57.035190 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:57.030027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:58747)
0620 08:04:57.030220 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:57.030231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:57.030281 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:57.030463 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:57.030463 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:57.030637 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:57.030685 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:57.030687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.031427 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.031428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:57.031968 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:57.031970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.032436 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.032437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:57.032500 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:57.033002 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:57.033007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:57.035086 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20260620 08:04:57.035244 8953 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:57.030154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:58747 (local address 127.31.115.254:37799)
0620 08:04:57.030260 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:57.030262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:57.030274 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:04:57.030362 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:57.030362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:57.030375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:57.030455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:57.030771 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.031340 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.032082 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.032348 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.032434 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:57.035178 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:58747: BlockingRecv error: recv got EOF from 127.31.115.194:58747 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 08:04:57 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:57 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:57.173166 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:57.168030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:55085)
0620 08:04:57.168239 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:57.168252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:57.168325 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:57.168484 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:57.168485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:57.168684 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:57.168730 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:57.168732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.169406 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.169408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:57.170045 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:57.170048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.170419 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.170420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:57.170478 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:57.171001 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:57.171006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:57.173062 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
W20260620 08:04:57.173228 8953 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:57.168149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:55085 (local address 127.31.115.254:37799)
0620 08:04:57.168240 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:57.168242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:57.168304 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:04:57.168383 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:57.168385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:57.168400 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:57.168476 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:57.168822 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.169318 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.170141 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.170342 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.170421 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:57.173165 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:55085: BlockingRecv error: recv got EOF from 127.31.115.195:55085 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 08:04:57 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:57 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:57.926290 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:57.917692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:54999)
0620 08:04:57.917863 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:57.917875 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:57.917939 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:57.918130 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:57.918132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:57.918345 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:57.918426 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:57.918428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.919306 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.919307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:57.919875 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:57.919877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.920321 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.920322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:57.920377 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:57.924228 (+ 3851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:57.924234 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:57.926225 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
W20260620 08:04:57.926486 8957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:57.917785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:54999 (local address 127.31.115.254:37799)
0620 08:04:57.917907 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:57.917910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:57.917924 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:04:57.917995 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:57.917996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:57.918008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:57.918111 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:57.918518 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.919196 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.919974 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:57.920222 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:57.920289 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:57.926409 (+ 6120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:54999: BlockingRecv error: recv got EOF from 127.31.115.193:54999 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
Jun 20 08:04:58 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:58 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:58.041047 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:58.035753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:50841)
0620 08:04:58.035898 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:58.035909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:58.035973 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:58.036125 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:58.036126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:58.036289 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:58.036336 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:58.036338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.037036 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.037037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:58.037666 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:58.037670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.038164 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.038165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:58.038223 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:58.038812 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:58.038818 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:58.040967 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260620 08:04:58.041162 8957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:58.035840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:50841 (local address 127.31.115.254:37799)
0620 08:04:58.035934 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:58.035936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:58.035991 (+ 55us) server_negotiation.cc:407] Connection header received
0620 08:04:58.036018 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:58.036019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:58.036030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:58.036086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:58.036433 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.036960 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.037779 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.037999 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.038085 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:58.041086 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:50841: BlockingRecv error: recv got EOF from 127.31.115.194:50841 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 08:04:58 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:58 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.195@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:58.181572 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:58.173893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.195:56409)
0620 08:04:58.174115 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:58.174127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:58.174228 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:58.174468 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:58.174469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:58.174907 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:58.174963 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:58.174966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.176174 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.176175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:58.177246 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:58.177249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.177934 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.177935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:58.177983 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:58.178500 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:58.178506 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:58.181451 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
W20260620 08:04:58.181628 8957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:58.173964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:56409 (local address 127.31.115.254:37799)
0620 08:04:58.174045 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:58.174047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:58.174229 (+ 182us) server_negotiation.cc:407] Connection header received
0620 08:04:58.174294 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:58.174296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:58.174316 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:58.174417 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:58.175047 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.175861 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.177558 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.177872 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.177931 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:58.181536 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.195:56409: BlockingRecv error: recv got EOF from 127.31.115.195:56409 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 08:04:58 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:58 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.193@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:58.937445 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:58.926889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.193:36751)
0620 08:04:58.927028 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:58.927044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:58.927093 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:58.927258 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:58.927259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:58.927421 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:58.927471 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:58.927472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.928255 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.928256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:58.928793 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:58.928795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.929257 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.929258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:58.929307 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:58.935117 (+ 5810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:58.935124 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:58.937374 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260620 08:04:58.937616 8961 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:58.926958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:36751 (local address 127.31.115.254:37799)
0620 08:04:58.927125 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:58.927127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:58.927136 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:04:58.927167 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:58.927168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:58.927178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:58.927243 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:58.927590 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.928149 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.928915 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:58.929172 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:58.929238 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:58.937503 (+ 8265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.193:36751: BlockingRecv error: recv got EOF from 127.31.115.193:36751 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
Jun 20 08:04:59 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:04:59 dist-test-slave-c4w8 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.115.194@KRBTEST.COM for oryx/127.31.115.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:04:59.046824 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.041866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:37799 (local address 127.31.115.194:35707)
0620 08:04:59.042071 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:59.042091 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:59.042160 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:59.042307 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:59.042308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:59.042482 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:59.042536 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.042538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.043227 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.043228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.043779 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.043782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.044191 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.044192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.044242 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.044797 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:04:59.044803 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:59.046745 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.115.254:37799: Server oryx/127.31.115.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
W20260620 08:04:59.046990 8961 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:59.041966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:35707 (local address 127.31.115.254:37799)
0620 08:04:59.042081 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.042084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.042162 (+ 78us) server_negotiation.cc:407] Connection header received
0620 08:04:59.042215 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.042216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.042228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.042283 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.042615 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.043119 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.043879 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.044098 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.044157 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.046906 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.115.194:35707: BlockingRecv error: recv got EOF from 127.31.115.194:35707 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 08:04:59.079511 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8485
I20260620 08:04:59.084403 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8637
I20260620 08:04:59.089027 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8770
I20260620 08:04:59.093451 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8415
2026-06-20T08:04:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17500 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T08:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:59Z Disabled control of system clock
I20260620 08:04:59.122428 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38909
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:38909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 08:04:59.206599 8966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:59.206825 8966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:59.206846 8966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:59.208006 8966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:59.208045 8966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:59.208058 8966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:59.208071 8966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:59.208083 8966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:59.209394 8966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:38909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:59.209694 8966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:59.209987 8966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:59.214967 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.214936 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.214946 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.215189 8966 server_base.cc:1034] running on GCE node
I20260620 08:04:59.215571 8966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:59.215942 8966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:59.217131 8966 hybrid_clock.cc:648] HybridClock initialized: now 1781942699217118 us; error 34 us; skew 500 ppm
I20260620 08:04:59.218665 8966 webserver.cc:460] Webserver started at http://127.31.115.254:46569/ using document root <none> and password file <none>
I20260620 08:04:59.218971 8966 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:59.219049 8966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:59.219213 8966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:59.220345 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "2f1c789956c943aeaa0bbd7b4110e364"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "c64cb316aaf68d41745a2cf3a02ec353"
server_key_iv: "bb614d1de4c26d7f942f08edd9becc52"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.220764 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "2f1c789956c943aeaa0bbd7b4110e364"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "c64cb316aaf68d41745a2cf3a02ec353"
server_key_iv: "bb614d1de4c26d7f942f08edd9becc52"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.222326 8966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:04:59.223315 8980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.223536 8966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:59.223695 8966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "2f1c789956c943aeaa0bbd7b4110e364"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "c64cb316aaf68d41745a2cf3a02ec353"
server_key_iv: "bb614d1de4c26d7f942f08edd9becc52"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.223781 8966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:59.245159 8966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:59.245520 8966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:59.245705 8966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:59.250157 8966 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:38909
I20260620 08:04:59.250213 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:38909 every 8 connection(s)
I20260620 08:04:59.250573 8966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:59.251602 9033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:59.254105 9033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364: Bootstrap starting.
I20260620 08:04:59.254734 9033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:59.255034 9033 log.cc:824] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:59.255909 9033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364: No bootstrap required, opened a new log
I20260620 08:04:59.257242 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 8966
I20260620 08:04:59.257368 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:59.257565 32207 external_mini_cluster.cc:1428] Setting key ec66993c80dca76b5e7006d98a04e979
I20260620 08:04:59.258013 9033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1c789956c943aeaa0bbd7b4110e364" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38909 } }
I20260620 08:04:59.258258 9033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:59.258347 9033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1c789956c943aeaa0bbd7b4110e364, State: Initialized, Role: FOLLOWER
I20260620 08:04:59.258551 9033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1c789956c943aeaa0bbd7b4110e364" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38909 } }
I20260620 08:04:59.258627 9033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:59.258652 9033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:59.258702 9033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:59.259323 9033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1c789956c943aeaa0bbd7b4110e364" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38909 } }
I20260620 08:04:59.259416 9033 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1c789956c943aeaa0bbd7b4110e364; no voters:
I20260620 08:04:59.259649 9033 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:59.259681 9038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:59.259804 9038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1c789956c943aeaa0bbd7b4110e364, State: Running, Role: LEADER
I20260620 08:04:59.259948 9033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:59.259954 9038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1c789956c943aeaa0bbd7b4110e364" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38909 } }
I20260620 08:04:59.260356 9038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1c789956c943aeaa0bbd7b4110e364. Latest consensus state: current_term: 1 leader_uuid: "2f1c789956c943aeaa0bbd7b4110e364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1c789956c943aeaa0bbd7b4110e364" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38909 } } }
I20260620 08:04:59.260447 9040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1c789956c943aeaa0bbd7b4110e364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1c789956c943aeaa0bbd7b4110e364" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38909 } } }
I20260620 08:04:59.260514 9038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:59.260528 9040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364 [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:59.260799 9045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:59.261461 9045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:59.263545 9045 catalog_manager.cc:1347] Generated new cluster ID: 78988758b8bb4f23bfa3676417735deb
I20260620 08:04:59.263612 9045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:59.264199 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.258175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.31.115.254:38909)
0620 08:04:59.258415 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.258425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.258459 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:04:59.258580 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.258594 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.258694 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.258841 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.259059 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.260627 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.261773 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.262383 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.262508 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.263386 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:59.263390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:59.263390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:59.263642 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:59.263692 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:59.263709 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:59.263786 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:59.263899 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 08:04:59.263983 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260620 08:04:59.275108 9045 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:59.275725 9045 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:59.280721 9045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1c789956c943aeaa0bbd7b4110e364: Generated new TSK 0
I20260620 08:04:59.280946 9045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:59.288467 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38909
--builtin_ntp_servers=127.31.115.212:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 08:04:59.378829 9057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:59.379046 9057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:59.379069 9057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:59.380308 9057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:59.380362 9057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:59.380389 9057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:59.382102 9057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:59.382383 9057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:59.382727 9057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:59.389209 9063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.389354 9062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.389209 9065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.389662 9057 server_base.cc:1034] running on GCE node
I20260620 08:04:59.389882 9057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:59.390170 9057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:59.391369 9057 hybrid_clock.cc:648] HybridClock initialized: now 1781942699391352 us; error 27 us; skew 500 ppm
I20260620 08:04:59.393038 9057 webserver.cc:460] Webserver started at http://127.31.115.193:37335/ using document root <none> and password file <none>
I20260620 08:04:59.393347 9057 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:59.393409 9057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:59.393558 9057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:59.394799 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "7c8144ab2f6e43d7aeacec9f2e3ad314"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "46e58cec4492297d5286341e2a2786f3"
server_key_iv: "3768d598d3f307018429a540b7330481"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.395169 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "7c8144ab2f6e43d7aeacec9f2e3ad314"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "46e58cec4492297d5286341e2a2786f3"
server_key_iv: "3768d598d3f307018429a540b7330481"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.396670 9057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:04:59.397579 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.397850 9057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:59.397959 9057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "7c8144ab2f6e43d7aeacec9f2e3ad314"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "46e58cec4492297d5286341e2a2786f3"
server_key_iv: "3768d598d3f307018429a540b7330481"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.398044 9057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:59.406503 9057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:59.406845 9057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:59.407008 9057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:59.407258 9057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:59.407624 9057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:59.407681 9057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.407722 9057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:59.407752 9057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.413646 9057 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:41251
I20260620 08:04:59.413717 9184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:41251 every 8 connection(s)
I20260620 08:04:59.414076 9057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:04:59.416949 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9057
I20260620 08:04:59.417026 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:04:59.417243 32207 external_mini_cluster.cc:1428] Setting key 6ccfa6c66eb8035778ac1e34000dacd9
I20260620 08:04:59.418509 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38909
--builtin_ntp_servers=127.31.115.212:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 08:04:59.420032 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.414882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:44819 (local address 127.31.115.254:38909)
0620 08:04:59.414947 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.414949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.415215 (+ 266us) server_negotiation.cc:407] Connection header received
0620 08:04:59.415787 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.415788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.415806 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.415857 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.416533 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.417164 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.418452 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.418723 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.418795 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.419725 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:59.419726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:59.419727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:59.419732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:59.419747 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:59.419751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:59.419804 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:59.419912 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 08:04:59.419951 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:04:59.420277 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.414904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:38909 (local address 127.31.115.193:44819)
0620 08:04:59.415092 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:59.415127 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:59.415665 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:59.415929 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:59.415933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:59.415942 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:59.416433 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.416436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.417263 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.417264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.418355 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.418358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.418810 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.418811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.418962 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.419535 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:59.419536 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:59.419564 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:59.419588 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:59.419810 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:59.419817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:59.419821 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:59.419914 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:04:59.420025 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260620 08:04:59.420835 9185 heartbeater.cc:347] Connected to a master server at 127.31.115.254:38909
I20260620 08:04:59.420953 9185 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:59.421151 9185 heartbeater.cc:511] Master 127.31.115.254:38909 requested a full tablet report, sending...
I20260620 08:04:59.421821 8997 ts_manager.cc:194] Registered new tserver with Master: 7c8144ab2f6e43d7aeacec9f2e3ad314 (127.31.115.193:41251)
I20260620 08:04:59.422530 8997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.193:44819
W20260620 08:04:59.504640 9188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:59.504860 9188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:59.504884 9188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:59.506218 9188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:59.506269 9188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:59.506299 9188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:04:59.507774 9188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:59.508059 9188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:59.508399 9188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:59.513772 9196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.513751 9194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.513808 9188 server_base.cc:1034] running on GCE node
W20260620 08:04:59.513957 9193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.514250 9188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:59.514513 9188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:59.515723 9188 hybrid_clock.cc:648] HybridClock initialized: now 1781942699515716 us; error 70 us; skew 500 ppm
I20260620 08:04:59.517290 9188 webserver.cc:460] Webserver started at http://127.31.115.194:35107/ using document root <none> and password file <none>
I20260620 08:04:59.517566 9188 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:59.517659 9188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:59.517792 9188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:59.519646 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "7ac29bfc38fe4ca895a31c6e9b5a8127"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "9861b07edc2c5ae45b0a7c479776b931"
server_key_iv: "6e415133015de03e284cdd68840c71d6"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.520179 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac29bfc38fe4ca895a31c6e9b5a8127"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "9861b07edc2c5ae45b0a7c479776b931"
server_key_iv: "6e415133015de03e284cdd68840c71d6"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.521788 9188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 08:04:59.522740 9202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.522929 9188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:59.523042 9188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "7ac29bfc38fe4ca895a31c6e9b5a8127"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "9861b07edc2c5ae45b0a7c479776b931"
server_key_iv: "6e415133015de03e284cdd68840c71d6"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.523151 9188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:59.543045 9188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:59.543442 9188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:59.543617 9188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:59.543879 9188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:59.544253 9188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:59.544317 9188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.544379 9188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:59.544407 9188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.551003 9188 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:41813
I20260620 08:04:59.551056 9316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:41813 every 8 connection(s)
I20260620 08:04:59.551473 9188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:04:59.554831 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9188
I20260620 08:04:59.554926 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:04:59.555125 32207 external_mini_cluster.cc:1428] Setting key b24b9a54f60670ce7120566dbd5c931b
I20260620 08:04:59.556236 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38909
--builtin_ntp_servers=127.31.115.212:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 08:04:59.558653 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.552230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:47367 (local address 127.31.115.254:38909)
0620 08:04:59.552314 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.552317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.552567 (+ 250us) server_negotiation.cc:407] Connection header received
0620 08:04:59.553172 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.553173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.553184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.553239 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.554168 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.555023 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.556379 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.556729 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.556828 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.558220 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:59.558222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:59.558223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:59.558230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:59.558267 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:59.558273 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:59.558339 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:59.558494 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:04:59.558528 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:04:59.559120 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.552235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:38909 (local address 127.31.115.194:47367)
0620 08:04:59.552450 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:59.552490 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:59.553064 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:59.553320 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:59.553324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:59.553333 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:59.554018 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.554023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.555142 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.555143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.556280 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.556283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.556933 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.556934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.557161 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.558011 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:59.558012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:59.558066 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:59.558113 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:59.558369 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:59.558375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:59.558379 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:04:59.558496 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 08:04:59.558690 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260620 08:04:59.561694 9317 heartbeater.cc:347] Connected to a master server at 127.31.115.254:38909
I20260620 08:04:59.561889 9317 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:59.562135 9317 heartbeater.cc:511] Master 127.31.115.254:38909 requested a full tablet report, sending...
I20260620 08:04:59.562585 8997 ts_manager.cc:194] Registered new tserver with Master: 7ac29bfc38fe4ca895a31c6e9b5a8127 (127.31.115.194:41813)
I20260620 08:04:59.563283 8997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.194:47367
W20260620 08:04:59.641499 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:59.641808 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:59.641834 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:59.642975 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:59.643014 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:59.643040 9320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:04:59.644359 9320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:38909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:59.644577 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:59.644845 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:59.650182 9325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.650182 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.650202 9320 server_base.cc:1034] running on GCE node
W20260620 08:04:59.650187 9328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.650707 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:59.650945 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:59.652110 9320 hybrid_clock.cc:648] HybridClock initialized: now 1781942699652075 us; error 61 us; skew 500 ppm
I20260620 08:04:59.653488 9320 webserver.cc:460] Webserver started at http://127.31.115.195:39997/ using document root <none> and password file <none>
I20260620 08:04:59.653867 9320 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:59.653918 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:59.654069 9320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:59.655197 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "fea096a2d9a54d5d9c1b5dd0b0582df9"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "75ae46f47f9a8046116d3ee8b1790fe4"
server_key_iv: "27109f97a1c35d4f0d7c60fc0321b736"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.655570 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "fea096a2d9a54d5d9c1b5dd0b0582df9"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "75ae46f47f9a8046116d3ee8b1790fe4"
server_key_iv: "27109f97a1c35d4f0d7c60fc0321b736"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.656975 9320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:04:59.657847 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.658031 9320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:59.658121 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "fea096a2d9a54d5d9c1b5dd0b0582df9"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "75ae46f47f9a8046116d3ee8b1790fe4"
server_key_iv: "27109f97a1c35d4f0d7c60fc0321b736"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.658179 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:59.688050 9320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:59.688398 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:59.688519 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:59.688776 9320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:04:59.689208 9320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:04:59.689249 9320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.689276 9320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:04:59.689321 9320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.695564 9320 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41499
I20260620 08:04:59.695600 9447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41499 every 8 connection(s)
I20260620 08:04:59.695981 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:04:59.702219 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.696872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:45579 (local address 127.31.115.254:38909)
0620 08:04:59.696960 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.696962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.697211 (+ 249us) server_negotiation.cc:407] Connection header received
0620 08:04:59.697931 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.697932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.697948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.698000 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.698790 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.699364 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.700603 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.700821 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.700900 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.701852 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:59.701854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:59.701854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:59.701860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:59.701877 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:59.701882 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:59.701955 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:59.702060 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 08:04:59.702093 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:04:59.702432 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.696862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:38909 (local address 127.31.115.195:45579)
0620 08:04:59.697055 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:04:59.697106 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:04:59.697806 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:04:59.698102 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:04:59.698106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:04:59.698115 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:04:59.698678 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.698683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.699475 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.699476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.700502 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:04:59.700505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:04:59.700920 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:04:59.700921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:04:59.701052 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:04:59.701691 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:04:59.701691 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:04:59.701722 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:04:59.701744 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:04:59.701955 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:04:59.701962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:04:59.701965 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:04:59.702048 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:04:59.702153 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260620 08:04:59.703121 9448 heartbeater.cc:347] Connected to a master server at 127.31.115.254:38909
I20260620 08:04:59.703271 9448 heartbeater.cc:464] Registering TS with master...
I20260620 08:04:59.703435 9448 heartbeater.cc:511] Master 127.31.115.254:38909 requested a full tablet report, sending...
I20260620 08:04:59.703442 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9320
I20260620 08:04:59.703511 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:04:59.703773 32207 external_mini_cluster.cc:1428] Setting key 5f846cde55b0aa6c3b4714c29b5325ce
I20260620 08:04:59.703859 8997 ts_manager.cc:194] Registered new tserver with Master: fea096a2d9a54d5d9c1b5dd0b0582df9 (127.31.115.195:41499)
I20260620 08:04:59.704299 8997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.195:45579
I20260620 08:04:59.704890 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:04:59.706920 9037 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:04:59.706351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.31.115.254:38909)
0620 08:04:59.706490 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.706497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.706563 (+ 66us) server_negotiation.cc:407] Connection header received
0620 08:04:59.706619 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.706620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.706631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.706682 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.706831 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58656: BlockingRecv error: recv got EOF from 127.0.0.1:58656 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 08:04:59.706943 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9057
I20260620 08:04:59.713348 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9188
I20260620 08:04:59.718253 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9320
I20260620 08:04:59.723443 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 8966
2026-06-20T08:04:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (629 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T08:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:04:59Z Disabled control of system clock
I20260620 08:04:59.752652 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41515
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:04:59.836529 9459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:59.836773 9459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:59.836797 9459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:59.838045 9459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:04:59.838090 9459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:59.838104 9459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:59.838117 9459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:04:59.838130 9459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:04:59.839534 9459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:59.839818 9459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:59.840157 9459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:59.845301 9467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.845307 9465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.845364 9464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.845494 9459 server_base.cc:1034] running on GCE node
I20260620 08:04:59.845777 9459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:04:59.846127 9459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:04:59.847294 9459 hybrid_clock.cc:648] HybridClock initialized: now 1781942699847280 us; error 28 us; skew 500 ppm
I20260620 08:04:59.848631 9459 webserver.cc:460] Webserver started at http://127.31.115.254:41385/ using document root <none> and password file <none>
I20260620 08:04:59.848898 9459 fs_manager.cc:359] Metadata directory not provided
I20260620 08:04:59.848946 9459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:04:59.849100 9459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:04:59.850286 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "e451ad3713af4c1eaa2618386fdaa4cc"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "cfcd59902f9dd73d904e59050b35cee8"
server_key_iv: "d917068a2d3c0c76f6b9336281f240b9"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.850660 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "e451ad3713af4c1eaa2618386fdaa4cc"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "cfcd59902f9dd73d904e59050b35cee8"
server_key_iv: "d917068a2d3c0c76f6b9336281f240b9"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.852092 9459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 08:04:59.852941 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:04:59.853142 9459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:04:59.853251 9459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "e451ad3713af4c1eaa2618386fdaa4cc"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "cfcd59902f9dd73d904e59050b35cee8"
server_key_iv: "d917068a2d3c0c76f6b9336281f240b9"
server_key_version: "encryptionkey@0"
I20260620 08:04:59.853339 9459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:04:59.857013 9459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:04:59.857301 9459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:04:59.857467 9459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:04:59.861749 9459 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:41515
I20260620 08:04:59.861800 9525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:41515 every 8 connection(s)
I20260620 08:04:59.862144 9459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:04:59.862841 9526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:04:59.865772 9526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc: Bootstrap starting.
I20260620 08:04:59.866361 9526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc: Neither blocks nor log segments found. Creating new log.
I20260620 08:04:59.866604 9526 log.cc:824] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc: Log is configured to *not* fsync() on all Append() calls
I20260620 08:04:59.867287 9526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc: No bootstrap required, opened a new log
I20260620 08:04:59.867573 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9459
I20260620 08:04:59.867676 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:04:59.867882 32207 external_mini_cluster.cc:1428] Setting key e5e773ba05b7fd17ba64732f211fe4c2
I20260620 08:04:59.869465 9526 raft_consensus.cc:348] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41515 } }
I20260620 08:04:59.869663 9526 raft_consensus.cc:374] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:04:59.869714 9526 raft_consensus.cc:729] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e451ad3713af4c1eaa2618386fdaa4cc, State: Initialized, Role: FOLLOWER
I20260620 08:04:59.869843 9526 consensus_queue.cc:260] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41515 } }
I20260620 08:04:59.869932 9526 raft_consensus.cc:388] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:04:59.869967 9526 raft_consensus.cc:482] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:04:59.870020 9526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:04:59.871078 9526 raft_consensus.cc:504] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41515 } }
I20260620 08:04:59.871119 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:04:59.868504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.31.115.254:41515)
0620 08:04:59.868701 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 08:04:59.868708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:04:59.868733 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:04:59.869458 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:04:59.869466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:04:59.869539 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:04:59.869818 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:04:59.870600 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:04:59.870604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:04:59.870604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:04:59.870608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:04:59.870651 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:04:59.870662 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:04:59.870725 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:04:59.870819 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 08:04:59.870913 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 08:04:59.871181 9526 leader_election.cc:302] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e451ad3713af4c1eaa2618386fdaa4cc; no voters:
I20260620 08:04:59.871362 9526 leader_election.cc:288] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:04:59.871641 9531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:04:59.871680 9526 sys_catalog.cc:564] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:04:59.871758 9531 raft_consensus.cc:686] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [term 1 LEADER]: Becoming Leader. State: Replica: e451ad3713af4c1eaa2618386fdaa4cc, State: Running, Role: LEADER
I20260620 08:04:59.871847 9531 consensus_queue.cc:237] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41515 } }
I20260620 08:04:59.872231 9532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41515 } } }
I20260620 08:04:59.872326 9532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:59.872213 9533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e451ad3713af4c1eaa2618386fdaa4cc. Latest consensus state: current_term: 1 leader_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e451ad3713af4c1eaa2618386fdaa4cc" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41515 } } }
I20260620 08:04:59.872586 9533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc [sys.catalog]: This master's current role is: LEADER
I20260620 08:04:59.872553 9540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:04:59.873198 9540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:04:59.875097 9540 catalog_manager.cc:1347] Generated new cluster ID: 492879b33ca94ddbb77c0568613ed09e
I20260620 08:04:59.875159 9540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:04:59.886545 9540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:04:59.886767 9540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:04:59.894860 9540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e451ad3713af4c1eaa2618386fdaa4cc: Generated new TSK 0
I20260620 08:04:59.895119 9540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:04:59.899048 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41515
--builtin_ntp_servers=127.31.115.212:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:04:59.988545 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:04:59.988830 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:04:59.988875 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:04:59.990281 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:04:59.990353 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:04:59.990401 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:04:59.991889 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:04:59.992194 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:04:59.992524 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:04:59.998849 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.998867 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:04:59.998849 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:04:59.999845 9550 server_base.cc:1034] running on GCE node
I20260620 08:05:00.000094 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:00.000393 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:00.001637 9550 hybrid_clock.cc:648] HybridClock initialized: now 1781942700001565 us; error 60 us; skew 500 ppm
I20260620 08:05:00.003823 9550 webserver.cc:460] Webserver started at http://127.31.115.193:34067/ using document root <none> and password file <none>
I20260620 08:05:00.004132 9550 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:00.004215 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:00.004348 9550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:00.006765 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "49f55cc142dc4ed5ad1f120031fe670b"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "ccd4ec84adfd3794e6e217256d14bd3c"
server_key_iv: "bda2c3bd9e6579044f70a32d3d519deb"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.007288 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "49f55cc142dc4ed5ad1f120031fe670b"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "ccd4ec84adfd3794e6e217256d14bd3c"
server_key_iv: "bda2c3bd9e6579044f70a32d3d519deb"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.010116 9550 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 08:05:00.011463 9564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.012004 9550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:00.012152 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "49f55cc142dc4ed5ad1f120031fe670b"
format_stamp: "Formatted at 2026-06-20 08:04:59 on dist-test-slave-c4w8"
server_key: "ccd4ec84adfd3794e6e217256d14bd3c"
server_key_iv: "bda2c3bd9e6579044f70a32d3d519deb"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.012269 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:00.016247 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:00.016579 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:00.016749 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:00.016969 9550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:00.017313 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:00.017369 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.017411 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:00.017437 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.023931 9550 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:35015
I20260620 08:05:00.024149 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:35015 every 8 connection(s)
I20260620 08:05:00.024466 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:05:00.024988 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9550
I20260620 08:05:00.025105 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:05:00.025326 32207 external_mini_cluster.cc:1428] Setting key e6fec6ae87d71dbeccc83d0f473e9716
I20260620 08:05:00.026320 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41515
--builtin_ntp_servers=127.31.115.212:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 08:05:00.028260 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:00.025392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:46787 (local address 127.31.115.254:41515)
0620 08:05:00.025454 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:00.025457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:00.025815 (+ 358us) server_negotiation.cc:407] Connection header received
0620 08:05:00.026348 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:00.026350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:00.026359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:00.026412 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:00.027903 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:00.027905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:00.027906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:05:00.027912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:00.027932 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:05:00.027938 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:00.028033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:00.028154 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 08:05:00.028177 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:05:00.028497 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:00.025410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41515 (local address 127.31.115.193:46787)
0620 08:05:00.025668 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:00.025712 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:00.026229 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:00.026497 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:00.026502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:00.026512 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:00.027699 (+ 1187us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:05:00.027699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:00.027733 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:05:00.027770 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:00.028030 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:00.028038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:00.028042 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:05:00.028168 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 08:05:00.028245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":117,"threads_started":1}
I20260620 08:05:00.029098 9678 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41515
I20260620 08:05:00.029206 9678 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:00.029385 9678 heartbeater.cc:511] Master 127.31.115.254:41515 requested a full tablet report, sending...
I20260620 08:05:00.030000 9490 ts_manager.cc:194] Registered new tserver with Master: 49f55cc142dc4ed5ad1f120031fe670b (127.31.115.193:35015)
W20260620 08:05:00.113466 9681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:00.113771 9681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:00.113813 9681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:00.115087 9681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:00.115149 9681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:00.115193 9681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:05:00.116758 9681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:00.117039 9681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:00.117350 9681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:00.122545 9687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:00.122601 9689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:00.122545 9686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:00.122849 9681 server_base.cc:1034] running on GCE node
I20260620 08:05:00.123118 9681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:00.123441 9681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:00.124615 9681 hybrid_clock.cc:648] HybridClock initialized: now 1781942700124602 us; error 35 us; skew 500 ppm
I20260620 08:05:00.126080 9681 webserver.cc:460] Webserver started at http://127.31.115.194:41033/ using document root <none> and password file <none>
I20260620 08:05:00.126365 9681 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:00.126441 9681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:00.126590 9681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:00.127684 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "37ddfbb224e043ee949302c6e4f1cf7c"
format_stamp: "Formatted at 2026-06-20 08:05:00 on dist-test-slave-c4w8"
server_key: "e16d9c60f02156fe2ed81154510059af"
server_key_iv: "fcbc271c08b7e5a3295a93184c351e51"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.128041 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "37ddfbb224e043ee949302c6e4f1cf7c"
format_stamp: "Formatted at 2026-06-20 08:05:00 on dist-test-slave-c4w8"
server_key: "e16d9c60f02156fe2ed81154510059af"
server_key_iv: "fcbc271c08b7e5a3295a93184c351e51"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.129395 9681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:05:00.130278 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.130589 9681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:00.130702 9681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "37ddfbb224e043ee949302c6e4f1cf7c"
format_stamp: "Formatted at 2026-06-20 08:05:00 on dist-test-slave-c4w8"
server_key: "e16d9c60f02156fe2ed81154510059af"
server_key_iv: "fcbc271c08b7e5a3295a93184c351e51"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.130790 9681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:00.134229 9681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:00.134502 9681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:00.134646 9681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:00.134863 9681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:00.135170 9681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:00.135224 9681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.135262 9681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:00.135291 9681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.141215 9681 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:45653
I20260620 08:05:00.141266 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:45653 every 8 connection(s)
I20260620 08:05:00.141589 9681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:05:00.142537 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9681
I20260620 08:05:00.142638 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:05:00.142827 32207 external_mini_cluster.cc:1428] Setting key cb47b64ada0b7cd404f23b7e7b2a7385
I20260620 08:05:00.143915 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41515
--builtin_ntp_servers=127.31.115.212:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 08:05:00.145569 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:00.142439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:41313 (local address 127.31.115.254:41515)
0620 08:05:00.142552 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:00.142554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:00.143259 (+ 705us) server_negotiation.cc:407] Connection header received
0620 08:05:00.143859 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:00.143860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:00.143867 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:00.143946 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:00.145323 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:00.145325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:00.145326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:05:00.145331 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:00.145348 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:05:00.145353 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:00.145406 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:00.145477 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:05:00.145493 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 08:05:00.145793 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:00.142440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41515 (local address 127.31.115.194:41313)
0620 08:05:00.143085 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:00.143146 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:00.143754 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:00.144001 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:00.144004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:00.144013 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:00.145162 (+ 1149us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:05:00.145162 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:00.145195 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:05:00.145225 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:00.145404 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:00.145410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:00.145413 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:05:00.145489 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:05:00.145555 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":86,"threads_started":1}
I20260620 08:05:00.146333 9809 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41515
I20260620 08:05:00.146435 9809 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:00.146554 9809 heartbeater.cc:511] Master 127.31.115.254:41515 requested a full tablet report, sending...
I20260620 08:05:00.146911 9490 ts_manager.cc:194] Registered new tserver with Master: 37ddfbb224e043ee949302c6e4f1cf7c (127.31.115.194:45653)
W20260620 08:05:00.227583 9812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:00.227802 9812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:00.227823 9812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:00.229030 9812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:00.229076 9812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:00.229110 9812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:05:00.230494 9812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:00.230768 9812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:00.231069 9812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:00.236573 9817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:00.236805 9812 server_base.cc:1034] running on GCE node
W20260620 08:05:00.236586 9818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:00.236606 9820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:00.237197 9812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:00.237428 9812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:00.238830 9812 hybrid_clock.cc:648] HybridClock initialized: now 1781942700238774 us; error 64 us; skew 500 ppm
I20260620 08:05:00.240311 9812 webserver.cc:460] Webserver started at http://127.31.115.195:42451/ using document root <none> and password file <none>
I20260620 08:05:00.240584 9812 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:00.240670 9812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:00.240826 9812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:00.242063 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "ea2de66b5e4d4174b3987cc327a49ebf"
format_stamp: "Formatted at 2026-06-20 08:05:00 on dist-test-slave-c4w8"
server_key: "2be421abf15f742051667a9b20c6d104"
server_key_iv: "657398d0d557b655f2ef81270a1b5ee7"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.242429 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "ea2de66b5e4d4174b3987cc327a49ebf"
format_stamp: "Formatted at 2026-06-20 08:05:00 on dist-test-slave-c4w8"
server_key: "2be421abf15f742051667a9b20c6d104"
server_key_iv: "657398d0d557b655f2ef81270a1b5ee7"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.243845 9812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:05:00.244681 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.244886 9812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:00.244997 9812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "ea2de66b5e4d4174b3987cc327a49ebf"
format_stamp: "Formatted at 2026-06-20 08:05:00 on dist-test-slave-c4w8"
server_key: "2be421abf15f742051667a9b20c6d104"
server_key_iv: "657398d0d557b655f2ef81270a1b5ee7"
server_key_version: "encryptionkey@0"
I20260620 08:05:00.245075 9812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:00.248654 9812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:00.248947 9812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:00.249087 9812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:00.249317 9812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:00.249702 9812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:00.249751 9812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.249786 9812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:00.249809 9812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:00.255626 9812 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:35987
I20260620 08:05:00.255712 9939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:35987 every 8 connection(s)
I20260620 08:05:00.256085 9812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:05:00.259133 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:00.256929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:42491 (local address 127.31.115.254:41515)
0620 08:05:00.257010 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:00.257012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:00.257225 (+ 213us) server_negotiation.cc:407] Connection header received
0620 08:05:00.257772 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:00.257773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:00.257778 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:00.257827 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:00.258935 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:00.258937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:00.258937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:05:00.258941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:00.258955 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:05:00.258959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:00.259017 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:00.259065 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 08:05:00.259078 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:05:00.259374 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:00.256861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41515 (local address 127.31.115.195:42491)
0620 08:05:00.257071 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:00.257127 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:00.257581 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:00.257891 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:00.257894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:00.257902 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:00.258780 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:05:00.258781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:00.258810 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:05:00.258839 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:00.259016 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:00.259018 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:00.259020 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:05:00.259072 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:05:00.259153 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260620 08:05:00.259980 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9812
I20260620 08:05:00.260077 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:05:00.260021 9940 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41515
I20260620 08:05:00.260149 9940 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:00.260319 32207 external_mini_cluster.cc:1428] Setting key 01ce0b81db755e0a7b4c50b10aecfb2e
I20260620 08:05:00.260340 9940 heartbeater.cc:511] Master 127.31.115.254:41515 requested a full tablet report, sending...
I20260620 08:05:00.260780 9489 ts_manager.cc:194] Registered new tserver with Master: ea2de66b5e4d4174b3987cc327a49ebf (127.31.115.195:35987)
I20260620 08:05:00.261343 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:05:00.263309 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:05:00.262769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.31.115.254:41515)
0620 08:05:00.262826 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:00.262828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:00.262984 (+ 156us) server_negotiation.cc:407] Connection header received
0620 08:05:00.263052 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:00.263052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:00.263057 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:00.263103 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:00.263227 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43086: BlockingRecv error: recv got EOF from 127.0.0.1:43086 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:05:00.263401 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9550
I20260620 08:05:00.268880 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9681
I20260620 08:05:00.274369 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9812
I20260620 08:05:00.278806 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9459
2026-06-20T08:05:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:05:00 dist-test-slave-c4w8 krb5kdc[9950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:05:00 dist-test-slave-c4w8 krb5kdc[9950](info): set up 2 sockets
Jun 20 08:05:00 dist-test-slave-c4w8 krb5kdc[9950](info): commencing operation
krb5kdc: starting...
W20260620 08:05:01.351102 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:05:01 dist-test-slave-c4w8 krb5kdc[9950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:05:01.478093 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41443
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:01.566635 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:01.566852 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:01.566879 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:01.568130 9963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:05:01.568176 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:01.568189 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:01.568202 9963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:05:01.568215 9963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:05:01.569953 9963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:01.570241 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:01.570494 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:01.575780 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:01.575780 9971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:01.575801 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:01.576082 9963 server_base.cc:1034] running on GCE node
I20260620 08:05:01.576339 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:01.576670 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:01.577831 9963 hybrid_clock.cc:648] HybridClock initialized: now 1781942701577821 us; error 35 us; skew 500 ppm
Jun 20 08:05:01 dist-test-slave-c4w8 krb5kdc[9950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:01.580565 9963 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:05:01.581070 9963 webserver.cc:460] Webserver started at http://127.31.115.254:37357/ using document root <none> and password file <none>
I20260620 08:05:01.581319 9963 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:01.581393 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:01.581542 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:01.582685 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "e4432a88bd954bf8af88073056752cbb"
format_stamp: "Formatted at 2026-06-20 08:05:01 on dist-test-slave-c4w8"
server_key: "f2f7e3c9b08fd1caf3836e3e593b3875"
server_key_iv: "c3e883460c682e3927043017b278ae54"
server_key_version: "encryptionkey@0"
I20260620 08:05:01.583125 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "e4432a88bd954bf8af88073056752cbb"
format_stamp: "Formatted at 2026-06-20 08:05:01 on dist-test-slave-c4w8"
server_key: "f2f7e3c9b08fd1caf3836e3e593b3875"
server_key_iv: "c3e883460c682e3927043017b278ae54"
server_key_version: "encryptionkey@0"
I20260620 08:05:01.584712 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:05:01.585542 9978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:01.585749 9963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:01.585865 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "e4432a88bd954bf8af88073056752cbb"
format_stamp: "Formatted at 2026-06-20 08:05:01 on dist-test-slave-c4w8"
server_key: "f2f7e3c9b08fd1caf3836e3e593b3875"
server_key_iv: "c3e883460c682e3927043017b278ae54"
server_key_version: "encryptionkey@0"
I20260620 08:05:01.585942 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:01.614279 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:01.616778 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:01.617012 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:01.622438 9963 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:41443
I20260620 08:05:01.622481 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:41443 every 8 connection(s)
I20260620 08:05:01.622856 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:05:01.623775 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 9963
I20260620 08:05:01.623896 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:05:01.624030 10031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:01.624083 32207 external_mini_cluster.cc:1428] Setting key d8ddc9e39aa5fbe0d9a944147311125f
I20260620 08:05:01.628013 10031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb: Bootstrap starting.
I20260620 08:05:01.628702 10031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:01.629046 10031 log.cc:824] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:01.629927 10031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb: No bootstrap required, opened a new log
Jun 20 08:05:01 dist-test-slave-c4w8 krb5kdc[9950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:01.631963 10031 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4432a88bd954bf8af88073056752cbb" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41443 } }
I20260620 08:05:01.632135 10031 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:01.632185 10031 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4432a88bd954bf8af88073056752cbb, State: Initialized, Role: FOLLOWER
I20260620 08:05:01.632359 10031 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4432a88bd954bf8af88073056752cbb" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41443 } }
I20260620 08:05:01.632483 10031 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:05:01.632522 10031 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:05:01.632583 10031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:01.633351 10031 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4432a88bd954bf8af88073056752cbb" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41443 } }
I20260620 08:05:01.633479 10031 leader_election.cc:302] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4432a88bd954bf8af88073056752cbb; no voters:
I20260620 08:05:01.633716 10031 leader_election.cc:288] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:05:01.633771 10036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:05:01.633925 10036 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [term 1 LEADER]: Becoming Leader. State: Replica: e4432a88bd954bf8af88073056752cbb, State: Running, Role: LEADER
I20260620 08:05:01.634027 10036 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4432a88bd954bf8af88073056752cbb" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41443 } }
I20260620 08:05:01.634060 10031 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:05:01.634416 10036 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4432a88bd954bf8af88073056752cbb. Latest consensus state: current_term: 1 leader_uuid: "e4432a88bd954bf8af88073056752cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4432a88bd954bf8af88073056752cbb" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41443 } } }
I20260620 08:05:01.634492 10036 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [sys.catalog]: This master's current role is: LEADER
I20260620 08:05:01.634730 10040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:05:01.635238 10040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:05:01.635811 10037 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4432a88bd954bf8af88073056752cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4432a88bd954bf8af88073056752cbb" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41443 } } }
I20260620 08:05:01.635905 10037 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb [sys.catalog]: This master's current role is: LEADER
I20260620 08:05:01.636193 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:01.624642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.31.115.254:41443)
0620 08:05:01.624887 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:01.624896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:01.624928 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:05:01.625675 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:01.625685 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:01.625754 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:01.625911 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:01.626547 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:01.627935 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:01.628828 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:01.629426 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:01.629563 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:01.631999 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:01.632003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:01.632003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:01.632024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:01.634165 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:01.634744 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:01.634748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:01.634749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:01.634880 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:01.635162 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:01.635163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:01.635163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:01.635487 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:01.635673 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:01.635824 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:05:01.635979 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":157,"threads_started":1}
I20260620 08:05:01.637740 10040 catalog_manager.cc:1347] Generated new cluster ID: 0ed9f8b06ac2443898dd5f89c989d871
I20260620 08:05:01.637800 10040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:05:01.646481 10040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:05:01.647095 10040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:05:01.651925 10040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb: Generated new TSK 0
I20260620 08:05:01.652164 10040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:05:01.689400 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41443
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:01.790858 10057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:01.791098 10057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:01.791126 10057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:01.792479 10057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:01.792541 10057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:01.792577 10057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:05:01.794229 10057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:01.794523 10057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:01.794816 10057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:01.800459 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:01.800627 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:01.800478 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:01.800774 10057 server_base.cc:1034] running on GCE node
I20260620 08:05:01.800992 10057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:01.801280 10057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:01.802469 10057 hybrid_clock.cc:648] HybridClock initialized: now 1781942701802465 us; error 43 us; skew 500 ppm
Jun 20 08:05:01 dist-test-slave-c4w8 krb5kdc[9950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:01.805529 10057 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:05:01.806087 10057 webserver.cc:460] Webserver started at http://127.31.115.193:42773/ using document root <none> and password file <none>
I20260620 08:05:01.806335 10057 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:01.806406 10057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:01.806557 10057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:01.807700 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "830fe425957743edae2ce0e849f849f6"
format_stamp: "Formatted at 2026-06-20 08:05:01 on dist-test-slave-c4w8"
server_key: "40b0993df0fac9d31390c178a9eb0a3c"
server_key_iv: "843e50b3e50878086cdd637f2fbd61ca"
server_key_version: "encryptionkey@0"
I20260620 08:05:01.808357 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "830fe425957743edae2ce0e849f849f6"
format_stamp: "Formatted at 2026-06-20 08:05:01 on dist-test-slave-c4w8"
server_key: "40b0993df0fac9d31390c178a9eb0a3c"
server_key_iv: "843e50b3e50878086cdd637f2fbd61ca"
server_key_version: "encryptionkey@0"
I20260620 08:05:01.810355 10057 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 08:05:01.811694 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:01.812058 10057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:05:01.812212 10057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "830fe425957743edae2ce0e849f849f6"
format_stamp: "Formatted at 2026-06-20 08:05:01 on dist-test-slave-c4w8"
server_key: "40b0993df0fac9d31390c178a9eb0a3c"
server_key_iv: "843e50b3e50878086cdd637f2fbd61ca"
server_key_version: "encryptionkey@0"
I20260620 08:05:01.812474 10057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:01.848976 10057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:01.851933 10057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:01.852102 10057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:01.852378 10057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:01.852732 10057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:01.852787 10057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:01.852829 10057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:01.852854 10057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:01.861377 10057 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:34403
I20260620 08:05:01.861478 10185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:34403 every 8 connection(s)
I20260620 08:05:01.861851 10057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:05:01.866333 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 10057
I20260620 08:05:01.866468 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:05:01.866729 32207 external_mini_cluster.cc:1428] Setting key 6a9ab317dad0e3f939baeb5283c12016
Jun 20 08:05:01 dist-test-slave-c4w8 krb5kdc[9950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:01.875619 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:01.862955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:48165 (local address 127.31.115.254:41443)
0620 08:05:01.863026 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:01.863029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:01.863041 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:05:01.864155 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:01.864156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:01.864168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:01.864242 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:01.865456 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:01.866235 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:01.867709 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:01.867932 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:01.868022 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:01.871167 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:01.871169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:01.871170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:01.871181 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:01.873161 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:01.874111 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:01.874112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:01.874113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:01.874198 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:01.874690 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:01.874692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:01.874693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:01.874883 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:01.874979 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:01.875389 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 08:05:01.875498 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:05:01.875818 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:01.862667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41443 (local address 127.31.115.193:48165)
0620 08:05:01.862901 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:01.862962 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:01.863902 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:01.864423 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:01.864429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:01.865085 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:01.865308 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:01.865313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:01.866403 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:01.866404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:01.867572 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:01.867576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:01.868036 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:01.868037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:01.868201 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:01.869051 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:05:01.869069 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:01.871019 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:01.873308 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:01.873321 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:01.873324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:01.873777 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:01.874347 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:01.874349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:01.874350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:01.874566 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:01.875027 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:01.875038 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:01.875228 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 08:05:01.875343 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 08:05:01.875512 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260620 08:05:01.876694 10186 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41443
I20260620 08:05:01.877012 10186 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:01.877292 10186 heartbeater.cc:511] Master 127.31.115.254:41443 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:05:01.878047 9995 ts_manager.cc:194] Registered new tserver with Master: 830fe425957743edae2ce0e849f849f6 (127.31.115.193:34403)
I20260620 08:05:01.878722 9995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:48165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:05:01.903843 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41443
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:01.992394 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:01.992642 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:01.992669 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:01.994050 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:01.994103 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:01.994134 10191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:05:01.995797 10191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:01.996102 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:01.996402 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:02.002164 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:02.002311 10191 server_base.cc:1034] running on GCE node
W20260620 08:05:02.002395 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:02.002172 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:02.002748 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:02.003082 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:02.004242 10191 hybrid_clock.cc:648] HybridClock initialized: now 1781942702004225 us; error 39 us; skew 500 ppm
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[9950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:02.007735 10191 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:05:02.008378 10191 webserver.cc:460] Webserver started at http://127.31.115.194:39263/ using document root <none> and password file <none>
I20260620 08:05:02.008662 10191 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:02.008754 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:02.008922 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:02.010378 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "e0c66ada892c4f5ca6afe541e174bc41"
format_stamp: "Formatted at 2026-06-20 08:05:02 on dist-test-slave-c4w8"
server_key: "97f6457d0c6fe4dee9e456637c14c02b"
server_key_iv: "aeb96dbbbb345d281891d3c88110142b"
server_key_version: "encryptionkey@0"
I20260620 08:05:02.010799 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "e0c66ada892c4f5ca6afe541e174bc41"
format_stamp: "Formatted at 2026-06-20 08:05:02 on dist-test-slave-c4w8"
server_key: "97f6457d0c6fe4dee9e456637c14c02b"
server_key_iv: "aeb96dbbbb345d281891d3c88110142b"
server_key_version: "encryptionkey@0"
I20260620 08:05:02.012385 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:05:02.013381 10206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:02.013706 10191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:02.013821 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "e0c66ada892c4f5ca6afe541e174bc41"
format_stamp: "Formatted at 2026-06-20 08:05:02 on dist-test-slave-c4w8"
server_key: "97f6457d0c6fe4dee9e456637c14c02b"
server_key_iv: "aeb96dbbbb345d281891d3c88110142b"
server_key_version: "encryptionkey@0"
I20260620 08:05:02.013913 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:02.035753 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:02.038341 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:02.038596 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:02.038878 10191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:02.039232 10191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:02.039294 10191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:02.039341 10191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:02.039371 10191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:02.046013 10191 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:43691
I20260620 08:05:02.046095 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:43691 every 8 connection(s)
I20260620 08:05:02.046464 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:05:02.050575 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 10191
I20260620 08:05:02.050701 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:05:02.050920 32207 external_mini_cluster.cc:1428] Setting key bddc6f572645cef4c3ce7c49563eea01
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[9950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:02.058571 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.047344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:58495 (local address 127.31.115.254:41443)
0620 08:05:02.047434 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.047436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.047684 (+ 248us) server_negotiation.cc:407] Connection header received
0620 08:05:02.048182 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.048184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.048201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.048271 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:02.049354 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.050102 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.051908 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.052105 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.052173 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.054899 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:02.054900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:02.054901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:02.054911 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:02.056818 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:02.057328 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:02.057330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:02.057330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:02.057400 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:02.057758 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:02.057759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:02.057760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:02.057935 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:02.058029 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.058373 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.058463 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:05:02.058833 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.047288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41443 (local address 127.31.115.194:58495)
0620 08:05:02.047483 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.047531 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.048070 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.048359 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.048363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.049010 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:02.049233 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.049239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.050218 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.050219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.051783 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.051787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.052209 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.052211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.052379 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.052993 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:05:02.053005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:02.054768 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:02.056941 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:02.056952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:02.056954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:02.057238 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:02.057492 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:02.057493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:02.057493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:02.057680 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:02.058031 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:02.058046 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:02.058194 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.058310 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.058494 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260620 08:05:02.059494 10320 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41443
I20260620 08:05:02.059651 10320 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:02.059857 10320 heartbeater.cc:511] Master 127.31.115.254:41443 requested a full tablet report, sending...
I20260620 08:05:02.060310 9995 ts_manager.cc:194] Registered new tserver with Master: e0c66ada892c4f5ca6afe541e174bc41 (127.31.115.194:43691)
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:05:02.060844 9995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:58495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:05:02.081554 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41443
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:02.167042 10325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:02.167294 10325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:02.167321 10325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:02.168633 10325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:02.168689 10325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:02.168717 10325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:05:02.170394 10325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:02.170672 10325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:02.170925 10325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:02.176172 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:02.176173 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:02.176213 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:02.176542 10325 server_base.cc:1034] running on GCE node
I20260620 08:05:02.176771 10325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:02.177023 10325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:02.178192 10325 hybrid_clock.cc:648] HybridClock initialized: now 1781942702178155 us; error 49 us; skew 500 ppm
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[9950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:02.180958 10325 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:05:02.181397 10325 webserver.cc:460] Webserver started at http://127.31.115.195:46049/ using document root <none> and password file <none>
I20260620 08:05:02.181694 10325 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:02.181751 10325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:02.181890 10325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:05:02.183329 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "e5401fdc44d443d8830ddffe20e6b093"
format_stamp: "Formatted at 2026-06-20 08:05:02 on dist-test-slave-c4w8"
server_key: "58288f196322fae563afddb6006bbd7f"
server_key_iv: "4d18615f41d4a69628c7afa24f185e68"
server_key_version: "encryptionkey@0"
I20260620 08:05:02.183746 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "e5401fdc44d443d8830ddffe20e6b093"
format_stamp: "Formatted at 2026-06-20 08:05:02 on dist-test-slave-c4w8"
server_key: "58288f196322fae563afddb6006bbd7f"
server_key_iv: "4d18615f41d4a69628c7afa24f185e68"
server_key_version: "encryptionkey@0"
I20260620 08:05:02.185241 10325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:05:02.186264 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:02.186513 10325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:02.186658 10325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "e5401fdc44d443d8830ddffe20e6b093"
format_stamp: "Formatted at 2026-06-20 08:05:02 on dist-test-slave-c4w8"
server_key: "58288f196322fae563afddb6006bbd7f"
server_key_iv: "4d18615f41d4a69628c7afa24f185e68"
server_key_version: "encryptionkey@0"
I20260620 08:05:02.186753 10325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:02.210476 10325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:02.212981 10325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:02.213168 10325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:02.213429 10325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:02.213806 10325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:02.213867 10325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:02.213912 10325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:02.213940 10325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:02.220338 10325 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:44121
I20260620 08:05:02.220381 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:44121 every 8 connection(s)
I20260620 08:05:02.220743 10325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[9950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:02.227345 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 10325
I20260620 08:05:02.227464 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:05:02.227632 32207 external_mini_cluster.cc:1428] Setting key 7202a5334908d0cf4985f79c2a419755
I20260620 08:05:02.231128 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.221595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:43009 (local address 127.31.115.254:41443)
0620 08:05:02.221696 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.221699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.221955 (+ 256us) server_negotiation.cc:407] Connection header received
0620 08:05:02.222449 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.222449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.222460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.222515 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:02.223500 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.224000 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.225157 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.225341 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.225395 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.227923 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:02.227924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:02.227925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:02.227932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:02.229508 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:02.230037 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:02.230038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:02.230039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:02.230105 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:02.230446 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:02.230447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:02.230448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:02.230578 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:02.230638 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.230989 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.231061 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:05:02.231307 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.221566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41443 (local address 127.31.115.195:43009)
0620 08:05:02.221827 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.221865 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.222361 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.222576 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.222581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.223190 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:02.223397 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.223402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.224184 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.224185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.225062 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.225066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.225444 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.225445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.225591 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.226172 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:05:02.226181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:02.227799 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:02.229685 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:02.229693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:02.229695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:02.229960 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:02.230198 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:02.230199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:02.230199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:02.230337 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:02.230660 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:02.230666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:02.230797 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.230908 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.231048 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":178,"threads_started":1}
I20260620 08:05:02.232075 10454 heartbeater.cc:347] Connected to a master server at 127.31.115.254:41443
I20260620 08:05:02.232179 10454 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:02.232435 10454 heartbeater.cc:511] Master 127.31.115.254:41443 requested a full tablet report, sending...
I20260620 08:05:02.232841 9995 ts_manager.cc:194] Registered new tserver with Master: e5401fdc44d443d8830ddffe20e6b093 (127.31.115.195:44121)
I20260620 08:05:02.233212 9995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:43009
I20260620 08:05:02.239566 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:05:02.247408 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.241167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.31.115.254:41443)
0620 08:05:02.241250 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.241253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.241351 (+ 98us) server_negotiation.cc:407] Connection header received
0620 08:05:02.241455 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.241456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.241473 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.241533 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:02.242137 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.242633 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.243396 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.243593 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.243692 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.244863 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:02.244865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:02.244865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:02.244873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:02.246198 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:02.246561 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:02.246562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:02.246563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:02.246622 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:02.246830 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:02.246831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:02.246831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:02.246990 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:02.247114 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.247214 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.247300 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:05:02.249210 9995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:05:02.250922 9995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:05:02.257871 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.253694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.31.115.195:44121)
0620 08:05:02.253901 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.253907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.253997 (+ 90us) server_negotiation.cc:407] Connection header received
0620 08:05:02.254067 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.254069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.254086 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.254162 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:02.254373 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.255294 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.256767 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.257407 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.257496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.257557 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.257730 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.257773 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
I20260620 08:05:02.257858 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.253495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44121 (local address 127.0.0.1:43214)
0620 08:05:02.253913 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.253925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.253991 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.254219 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.254222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.254224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:02.254306 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.254309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.255393 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.255394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.256678 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.256681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.257505 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.257506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.257593 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.257635 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.257714 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.257773 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":73,"threads_started":1}
I20260620 08:05:02.258368 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.253476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:43691 (local address 127.0.0.1:53768)
0620 08:05:02.253715 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.253760 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.253835 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.254823 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.254825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.254827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:02.254876 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.254877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.256032 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.256033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.257050 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.257053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.258122 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.258123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.258180 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.258183 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.258229 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.258269 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260620 08:05:02.258396 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.253733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:34403 (local address 127.0.0.1:50646)
0620 08:05:02.253949 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.253963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.254021 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.254422 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.254423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.254425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:02.254468 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.254469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.255729 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.255730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.256778 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.256779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.257682 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.257683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.257745 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.257750 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.257802 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.257846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260620 08:05:02.258729 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.253563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.31.115.194:43691)
0620 08:05:02.254610 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.254614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.254622 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:05:02.254704 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.254707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.254725 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.254809 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:02.254960 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.255929 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.257160 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.257934 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.258005 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.258080 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.258598 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.258645 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":162,"threads_started":1}
I20260620 08:05:02.258750 10388 tablet_service.cc:1467] Processing CreateTablet for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e (DEFAULT_TABLE table=test-table [id=0de1ac36cef14b079ca8c55425697e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:05:02.258759 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.253954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.31.115.193:34403)
0620 08:05:02.254163 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.254168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.254177 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:05:02.254219 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.254221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.254257 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.254338 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:02.254527 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.255618 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.256846 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.257487 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.257559 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.257645 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.257907 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.258369 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260620 08:05:02.259176 10388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:02.259486 10254 tablet_service.cc:1467] Processing CreateTablet for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e (DEFAULT_TABLE table=test-table [id=0de1ac36cef14b079ca8c55425697e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:05:02.259712 10120 tablet_service.cc:1467] Processing CreateTablet for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e (DEFAULT_TABLE table=test-table [id=0de1ac36cef14b079ca8c55425697e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:05:02.259827 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:02.260052 10120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:02.262766 10471 tablet_bootstrap.cc:492] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6: Bootstrap starting.
I20260620 08:05:02.263147 10472 tablet_bootstrap.cc:492] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Bootstrap starting.
I20260620 08:05:02.263161 10473 tablet_bootstrap.cc:492] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Bootstrap starting.
I20260620 08:05:02.263748 10473 tablet_bootstrap.cc:654] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:02.263785 10472 tablet_bootstrap.cc:654] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:02.263763 10471 tablet_bootstrap.cc:654] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:02.264266 10473 log.cc:824] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:02.264405 10472 log.cc:824] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:02.264428 10471 log.cc:824] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:02.265224 10472 tablet_bootstrap.cc:492] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: No bootstrap required, opened a new log
I20260620 08:05:02.265250 10471 tablet_bootstrap.cc:492] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6: No bootstrap required, opened a new log
I20260620 08:05:02.265302 10472 ts_tablet_manager.cc:1397] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:05:02.265313 10471 ts_tablet_manager.cc:1397] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:05:02.265347 10473 tablet_bootstrap.cc:492] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: No bootstrap required, opened a new log
I20260620 08:05:02.265407 10473 ts_tablet_manager.cc:1397] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:05:02.266819 10472 raft_consensus.cc:348] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.266999 10472 raft_consensus.cc:374] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:02.267045 10472 raft_consensus.cc:729] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5401fdc44d443d8830ddffe20e6b093, State: Initialized, Role: FOLLOWER
I20260620 08:05:02.267179 10472 consensus_queue.cc:260] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.267406 10471 raft_consensus.cc:348] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.267509 10472 ts_tablet_manager.cc:1428] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:05:02.267406 10473 raft_consensus.cc:348] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.267577 10471 raft_consensus.cc:374] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:02.267577 10473 raft_consensus.cc:374] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:02.267625 10473 raft_consensus.cc:729] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c66ada892c4f5ca6afe541e174bc41, State: Initialized, Role: FOLLOWER
I20260620 08:05:02.267627 10471 raft_consensus.cc:729] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830fe425957743edae2ce0e849f849f6, State: Initialized, Role: FOLLOWER
I20260620 08:05:02.267521 10454 heartbeater.cc:503] Master 127.31.115.254:41443 was elected leader, sending a full tablet report...
I20260620 08:05:02.267747 10473 consensus_queue.cc:260] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.267779 10471 consensus_queue.cc:260] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.268069 10186 heartbeater.cc:503] Master 127.31.115.254:41443 was elected leader, sending a full tablet report...
I20260620 08:05:02.268308 10471 ts_tablet_manager.cc:1428] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:05:02.268527 10473 ts_tablet_manager.cc:1428] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:05:02.268680 10320 heartbeater.cc:503] Master 127.31.115.254:41443 was elected leader, sending a full tablet report...
W20260620 08:05:02.297135 10321 tablet.cc:2369] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:05:02.335274 10478 raft_consensus.cc:482] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:05:02.335418 10478 raft_consensus.cc:504] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.335933 10478 leader_election.cc:288] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0c66ada892c4f5ca6afe541e174bc41 (127.31.115.194:43691), e5401fdc44d443d8830ddffe20e6b093 (127.31.115.195:44121)
I20260620 08:05:02.340198 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.336238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:60827 (local address 127.31.115.194:43691)
0620 08:05:02.336290 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.336292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.336428 (+ 136us) server_negotiation.cc:407] Connection header received
0620 08:05:02.336582 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.336583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.336591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.336634 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:02.336796 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.337484 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.339084 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.339675 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.339760 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.339772 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.340034 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.340104 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:05:02.340220 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.336163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44121 (local address 127.31.115.193:44473)
0620 08:05:02.336224 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.336237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.336288 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.336502 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.336504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.336505 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:02.336588 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.336595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.337631 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.337632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.338964 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.338968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.339834 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.339835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.339943 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.339948 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.340015 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.340079 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":55}
I20260620 08:05:02.340268 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.336354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:44473 (local address 127.31.115.195:44121)
0620 08:05:02.336399 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.336401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.336409 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:05:02.336437 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.336442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.336450 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.336489 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:02.336687 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.337444 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.339084 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.339762 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.339829 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.339838 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.340145 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.340191 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:05:02.340588 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.336165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:43691 (local address 127.31.115.193:60827)
0620 08:05:02.336354 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.336369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:02.336444 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:02.336660 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:02.336661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:02.336663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:02.336727 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.336728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.337588 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.337589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.338964 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:02.338968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.339854 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.339855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:02.339944 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.339960 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:05:02.340013 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:05:02.340074 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":121,"thread_start_us":67,"threads_started":1}
I20260620 08:05:02.340656 10408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ef8ddf6bf4b5e83a492bf25f30d8e" candidate_uuid: "830fe425957743edae2ce0e849f849f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5401fdc44d443d8830ddffe20e6b093" is_pre_election: true
I20260620 08:05:02.340853 10408 raft_consensus.cc:2454] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830fe425957743edae2ce0e849f849f6 in term 0.
I20260620 08:05:02.340931 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ef8ddf6bf4b5e83a492bf25f30d8e" candidate_uuid: "830fe425957743edae2ce0e849f849f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c66ada892c4f5ca6afe541e174bc41" is_pre_election: true
I20260620 08:05:02.341087 10274 raft_consensus.cc:2454] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830fe425957743edae2ce0e849f849f6 in term 0.
I20260620 08:05:02.341162 10073 leader_election.cc:302] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830fe425957743edae2ce0e849f849f6, e5401fdc44d443d8830ddffe20e6b093; no voters:
I20260620 08:05:02.341312 10478 raft_consensus.cc:2781] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:05:02.341369 10478 raft_consensus.cc:482] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:05:02.341387 10478 raft_consensus.cc:3037] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:02.342213 10478 raft_consensus.cc:504] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.342374 10478 leader_election.cc:288] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [CANDIDATE]: Term 1 election: Requested vote from peers e0c66ada892c4f5ca6afe541e174bc41 (127.31.115.194:43691), e5401fdc44d443d8830ddffe20e6b093 (127.31.115.195:44121)
I20260620 08:05:02.342549 10274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ef8ddf6bf4b5e83a492bf25f30d8e" candidate_uuid: "830fe425957743edae2ce0e849f849f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c66ada892c4f5ca6afe541e174bc41"
I20260620 08:05:02.342617 10274 raft_consensus.cc:3037] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:02.343321 10274 raft_consensus.cc:2454] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830fe425957743edae2ce0e849f849f6 in term 1.
I20260620 08:05:02.343453 10408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3ef8ddf6bf4b5e83a492bf25f30d8e" candidate_uuid: "830fe425957743edae2ce0e849f849f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5401fdc44d443d8830ddffe20e6b093"
I20260620 08:05:02.343524 10076 leader_election.cc:302] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830fe425957743edae2ce0e849f849f6, e0c66ada892c4f5ca6afe541e174bc41; no voters:
I20260620 08:05:02.343542 10408 raft_consensus.cc:3037] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:02.343636 10478 raft_consensus.cc:2781] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:05:02.343823 10478 raft_consensus.cc:686] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [term 1 LEADER]: Becoming Leader. State: Replica: 830fe425957743edae2ce0e849f849f6, State: Running, Role: LEADER
I20260620 08:05:02.343923 10478 consensus_queue.cc:237] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } }
I20260620 08:05:02.344480 10408 raft_consensus.cc:2454] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830fe425957743edae2ce0e849f849f6 in term 1.
I20260620 08:05:02.344889 9994 catalog_manager.cc:5495] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830fe425957743edae2ce0e849f849f6 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "830fe425957743edae2ce0e849f849f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830fe425957743edae2ce0e849f849f6" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 34403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 } health_report { overall_health: UNKNOWN } } }
I20260620 08:05:02.351999 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.349490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.31.115.193:34403)
0620 08:05:02.349556 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:02.349559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:02.349568 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:05:02.349655 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:02.349656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:02.349673 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:02.349750 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:05:02.349892 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.350371 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.351186 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:02.351383 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:02.351477 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:02.351650 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:05:02.351750 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:05:02.351847 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:02.351867 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 08:05:02.351887 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:05:02.353933 10408 raft_consensus.cc:1261] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 1 FOLLOWER]: Refusing update from remote peer 830fe425957743edae2ce0e849f849f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:05:02.353958 10274 raft_consensus.cc:1261] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 1 FOLLOWER]: Refusing update from remote peer 830fe425957743edae2ce0e849f849f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:05:02.354316 10481 consensus_queue.cc:1035] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5401fdc44d443d8830ddffe20e6b093" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:05:02.354460 10478 consensus_queue.cc:1035] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P 830fe425957743edae2ce0e849f849f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c66ada892c4f5ca6afe541e174bc41" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:05:02.356823 10484 mvcc.cc:205] Tried to move back new op lower bound from 7298837308838543360 to 7298837308801515520. Current Snapshot: MvccSnapshot[applied={T|T < 7298837308838543360}]
I20260620 08:05:02.357926 10486 mvcc.cc:205] Tried to move back new op lower bound from 7298837308838543360 to 7298837308801515520. Current Snapshot: MvccSnapshot[applied={T|T < 7298837308838543360}]
I20260620 08:05:02.359170 9994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:05:02.359268 9994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:05:02.360083 9994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4432a88bd954bf8af88073056752cbb: Sending DeleteTablet for 3 replicas of tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e
I20260620 08:05:02.360420 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:05:02 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53768
I20260620 08:05:02.360488 10120 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:05:02 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50646
I20260620 08:05:02.360512 10388 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:05:02 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:43214
I20260620 08:05:02.360661 10494 tablet_replica.cc:330] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: stopping tablet replica
I20260620 08:05:02.360766 10494 raft_consensus.cc:2229] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:05:02.360868 10494 raft_consensus.cc:2258] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:05:02.361038 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 10057
I20260620 08:05:02.361672 10496 tablet_replica.cc:330] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: stopping tablet replica
I20260620 08:05:02.361776 10496 raft_consensus.cc:2229] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:05:02.361876 10496 raft_consensus.cc:2258] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:05:02.362383 10494 ts_tablet_manager.cc:1905] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:05:02.364341 10494 ts_tablet_manager.cc:1918] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:05:02.364439 10494 log.cc:1196] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/eb3ef8ddf6bf4b5e83a492bf25f30d8e
I20260620 08:05:02.364701 10494 ts_tablet_manager.cc:1939] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e0c66ada892c4f5ca6afe541e174bc41: Deleting consensus metadata
I20260620 08:05:02.365150 9982 catalog_manager.cc:4841] TS e0c66ada892c4f5ca6afe541e174bc41 (127.31.115.194:43691): tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e (table test-table [id=0de1ac36cef14b079ca8c55425697e1c]) successfully deleted
I20260620 08:05:02.365881 10496 ts_tablet_manager.cc:1905] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:05:02.367156 9982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:34403 (error 108)
I20260620 08:05:02.367336 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 10191
I20260620 08:05:02.367874 10496 ts_tablet_manager.cc:1918] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:05:02.368031 10496 log.cc:1196] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/eb3ef8ddf6bf4b5e83a492bf25f30d8e
I20260620 08:05:02.368197 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:02.368053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:34403 (local address 127.0.0.1:50662)
0620 08:05:02.368121 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:02.368165 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 08:05:02.368323 9982 catalog_manager.cc:4584] TS 830fe425957743edae2ce0e849f849f6 (127.31.115.193:34403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e: Network error: Client connection negotiation failed: client connection to 127.31.115.193:34403: connect: Connection refused (error 111)
I20260620 08:05:02.368364 10496 ts_tablet_manager.cc:1939] T eb3ef8ddf6bf4b5e83a492bf25f30d8e P e5401fdc44d443d8830ddffe20e6b093: Deleting consensus metadata
I20260620 08:05:02.368870 9979 catalog_manager.cc:4841] TS e5401fdc44d443d8830ddffe20e6b093 (127.31.115.195:44121): tablet eb3ef8ddf6bf4b5e83a492bf25f30d8e (table test-table [id=0de1ac36cef14b079ca8c55425697e1c]) successfully deleted
I20260620 08:05:02.373764 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 10325
I20260620 08:05:02.378793 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 9963
2026-06-20T08:05:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2100 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[10500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[10500](info): set up 2 sockets
Jun 20 08:05:02 dist-test-slave-c4w8 krb5kdc[10500](info): commencing operation
krb5kdc: starting...
W20260620 08:05:03.439330 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:05:03 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:05:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/rangeradmin_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/rangeradmin_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/rangerlookup_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/rangerlookup_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
I20260620 08:05:03.612207 32207 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvjmHcA/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/postgres -l logfile start
2026-06-20 08:05:06.540 UTC [10530] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 08:05:06.540 UTC [10530] LOG: listening on IPv4 address "127.31.115.212", port 51351
2026-06-20 08:05:06.544 UTC [10530] LOG: listening on Unix socket "/tmp/.s.PGSQL.51351"
2026-06-20 08:05:06.580 UTC [10533] LOG: database system was shut down at 2026-06-20 08:05:04 UTC
2026-06-20 08:05:06.585 UTC [10530] LOG: database system is ready to accept connections
I20260620 08:05:07.564723 32207 mini_postgres.cc:96] Postgres bound to 51351
2026-06-20 08:05:07.574 UTC [10541] FATAL: database "slave" does not exist
127.31.115.212:51351 - accepting connections
I20260620 08:05:07.574944 32207 mini_ranger.cc:161] Starting Ranger...
I20260620 08:05:07.592198 32207 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 08:05:08.151269 32207 mini_ranger.cc:87] Created ranger Postgres database
I20260620 08:05:08.151369 32207 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-admin
2026-06-20 08:05:08,712 [I] DB FLAVOR :POSTGRES
2026-06-20 08:05:08,712 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:05:08,713 [I] Checking connection..
2026-06-20 08:05:08,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 08:05:09,023 [I] Checking connection passed.
2026-06-20 08:05:09,024 [I] --------- Verifying version history table ---------
2026-06-20 08:05:09,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:05:09,280 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:05:09,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:05:09,539 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:05:09,539 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 08:05:09,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 08:05:09.812 UTC [10605] WARNING: there is no transaction in progress
2026-06-20 08:05:09,819 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 08:05:09,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:05:10,081 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 08:05:10,081 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:05:10,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 08:05:10,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 08:05:10,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c4w8.c.gcp-upstream.internal','N') ;"
2026-06-20 08:05:10,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:05:11,086 [I] Table x_portal_user does not exist in database ranger
2026-06-20 08:05:11,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:05:11,327 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 08:05:11,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:05:11,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:05:11,840 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 08:05:11,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 08:05:12.940 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:12.961 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:12.981 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.000 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.018 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.039 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.103 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.110 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.127 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.138 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.149 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.165 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.177 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.184 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.192 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:13.200 UTC [10713] WARNING: there is no transaction in progress
2026-06-20 08:05:14,013 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 08:05:14,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 08:05:14,290 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 08:05:14,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:05:14,556 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 08:05:14,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:05:14,839 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 08:05:14,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:05:15,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:05:15,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c4w8.c.gcp-upstream.internal';"
2026-06-20 08:05:15,610 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 08:05:15,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 08:05:15,853 [I] DB_PATCHES have already been applied
I20260620 08:05:15.859051 32207 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvjmHcA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 08:05:15.859193 32207 mini_ranger.cc:193] Using host: 127.31.115.212
I20260620 08:05:15.862888 32207 mini_ranger.cc:227] Ranger admin URL: http://127.31.115.212:50933
Jun 20, 2026 8:05:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:05:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:05:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:05:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:05:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:05:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:05:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 08:05:18.117134 32207 mini_ranger.cc:160] Time spent starting Ranger: real 10.542s user 0.001s sys 0.011s
Jun 20, 2026 8:05:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:05:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 8:05:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:05:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:05:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 08:05:43.818240 32207 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/rangerkms_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/rangerkms_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.115.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.115.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 08:05:43 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942743, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 08:05:43.916342 32207 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 08:05:43.930707 32207 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 08:05:45.171298 32207 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 08:05:45.171386 32207 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms
I20260620 08:05:45.171404 32207 mini_ranger_kms.cc:227] Using postgres at 127.31.115.212:51351
2026-06-20 08:05:45,875 [I] DB FLAVOR :POSTGRES
2026-06-20 08:05:45,875 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:05:45,876 [I] Checking connection
2026-06-20 08:05:45,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 08:05:46,162 [I] connection success
2026-06-20 08:05:46,162 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 08:05:46,162 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 08:05:46,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 08:05:46,449 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 08:05:46,449 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:05:46,449 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 08:05:46,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:51351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 08:05:46,746 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 08:05:47.310173 32207 mini_ranger_kms.cc:310] Created kms service
I20260620 08:05:47.520399 32207 mini_ranger_kms.cc:326] Created kudu user
I20260620 08:05:47.569600 32207 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 08:05:47 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942743, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:05:48.051672 32207 mini_ranger_kms.cc:384] Added ranger policy
I20260620 08:05:48.051801 32207 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 08:05:48.051827 32207 mini_ranger_kms.cc:240] Using host: 127.31.115.212
I20260620 08:05:48.056634 32207 mini_ranger_kms.cc:276] Ranger KMS PID: 10895
I20260620 08:05:48.056797 32207 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.115.212:41587
Jun 20, 2026 8:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 08:05:50.382434 32207 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.466s user 0.005s sys 0.261s
I20260620 08:05:50.382556 32207 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 08:05:50.382635 32207 mini_ranger_kms.cc:401] 127.31.115.212:41587/kms/v1/keys
Jun 20 08:05:50 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781942750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.115.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 08:05:51 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.115.212@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
Jun 20 08:05:51 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.115.212@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
Jun 20 08:05:54 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:05:54.066121 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:44065
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:44065
--ranger_config_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:41587/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:54.151825 10945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:54.152051 10945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:54.152077 10945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:54.153394 10945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:05:54.153446 10945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:54.153461 10945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:54.153473 10945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:05:54.153486 10945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:05:54.155260 10945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.115.212:41587/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:44065
--ranger_config_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:54.155524 10945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:54.155773 10945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:54.161908 10951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:54.161964 10945 server_base.cc:1034] running on GCE node
W20260620 08:05:54.161998 10950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:54.162135 10953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:54.162420 10945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:54.162779 10945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:54.163960 10945 hybrid_clock.cc:648] HybridClock initialized: now 1781942754163938 us; error 37 us; skew 500 ppm
Jun 20 08:05:54 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942754, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:54.171310 10945 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:05:54.171890 10945 webserver.cc:460] Webserver started at http://127.31.115.254:40671/ using document root <none> and password file <none>
I20260620 08:05:54.172134 10945 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:54.172197 10945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:54.172353 10945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:05:54 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942754, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:05:54.236642 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "701c1ed0659d4422a454e3da0d3463ec"
format_stamp: "Formatted at 2026-06-20 08:05:54 on dist-test-slave-c4w8"
server_key: "6c2c02c0ab31430223ab8f1a14cedbb2"
server_key_iv: "5cc51c5ba6382b81087534c79a7fc381"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:54.237073 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "701c1ed0659d4422a454e3da0d3463ec"
format_stamp: "Formatted at 2026-06-20 08:05:54 on dist-test-slave-c4w8"
server_key: "6c2c02c0ab31430223ab8f1a14cedbb2"
server_key_iv: "5cc51c5ba6382b81087534c79a7fc381"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:54.238739 10945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 08:05:54.251369 10960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:54.251652 10945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:05:54.251798 10945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "701c1ed0659d4422a454e3da0d3463ec"
format_stamp: "Formatted at 2026-06-20 08:05:54 on dist-test-slave-c4w8"
server_key: "6c2c02c0ab31430223ab8f1a14cedbb2"
server_key_iv: "5cc51c5ba6382b81087534c79a7fc381"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:54.251946 10945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:54.269816 10945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:54.272043 10945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:54.272327 10945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:54.277766 10945 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:44065
I20260620 08:05:54.277784 11012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:44065 every 8 connection(s)
I20260620 08:05:54.278254 10945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:05:54.278993 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:54.281919 11013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec: Bootstrap starting.
I20260620 08:05:54.282604 11013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:54.282764 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 10945
I20260620 08:05:54.282878 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:05:54.283023 11013 log.cc:824] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:54.283066 32207 external_mini_cluster.cc:1428] Setting key 460628ea811b69280981a5303ee4f198
I20260620 08:05:54.283903 11013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec: No bootstrap required, opened a new log
I20260620 08:05:54.286651 11013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c1ed0659d4422a454e3da0d3463ec" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44065 } }
I20260620 08:05:54.286828 11013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:54.286859 11013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701c1ed0659d4422a454e3da0d3463ec, State: Initialized, Role: FOLLOWER
I20260620 08:05:54.287051 11013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c1ed0659d4422a454e3da0d3463ec" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44065 } }
I20260620 08:05:54.287177 11013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:05:54.287230 11013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:05:54.287300 11013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:54.288031 11013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c1ed0659d4422a454e3da0d3463ec" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44065 } }
I20260620 08:05:54.288223 11013 leader_election.cc:302] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701c1ed0659d4422a454e3da0d3463ec; no voters:
I20260620 08:05:54.288407 11013 leader_election.cc:288] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:05:54.288453 11018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:05:54.288681 11018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [term 1 LEADER]: Becoming Leader. State: Replica: 701c1ed0659d4422a454e3da0d3463ec, State: Running, Role: LEADER
I20260620 08:05:54.288790 11013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:05:54.288810 11018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c1ed0659d4422a454e3da0d3463ec" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44065 } }
I20260620 08:05:54.289534 11018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701c1ed0659d4422a454e3da0d3463ec. Latest consensus state: current_term: 1 leader_uuid: "701c1ed0659d4422a454e3da0d3463ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c1ed0659d4422a454e3da0d3463ec" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44065 } } }
I20260620 08:05:54.289712 11018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [sys.catalog]: This master's current role is: LEADER
I20260620 08:05:54.289920 11020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "701c1ed0659d4422a454e3da0d3463ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c1ed0659d4422a454e3da0d3463ec" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 44065 } } }
I20260620 08:05:54.290068 11020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec [sys.catalog]: This master's current role is: LEADER
I20260620 08:05:54.290925 11013 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 08:05:54 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:54.293758 11023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:05:54.295861 11023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:05:54.300650 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:54.283657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.31.115.254:44065)
0620 08:05:54.283847 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:54.283857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:54.283947 (+ 90us) server_negotiation.cc:407] Connection header received
0620 08:05:54.284518 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:54.284527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:54.284593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:54.284752 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:54.285532 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:54.287037 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:54.288136 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:54.288783 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:54.289266 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:54.293087 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:54.293140 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:54.293140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:54.293260 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:54.298311 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:54.298815 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:54.298818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:54.298819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:54.298931 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:54.299146 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:54.299147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:54.299147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:54.299475 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:54.299625 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:54.300258 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 08:05:54.300418 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260620 08:05:54.306213 11023 catalog_manager.cc:1347] Generated new cluster ID: 7965873021e14431bf97f46c781a84e8
I20260620 08:05:54.306303 11023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:05:54.326287 11023 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:05:54.326898 11023 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:05:54.335059 11023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec: Generated new TSK 0
I20260620 08:05:54.335296 11023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:05:55.084424 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:41587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44065
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:55.192662 11058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:55.192862 11058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:55.192888 11058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:55.194123 11058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:55.194167 11058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:55.194196 11058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:05:55.195869 11058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.115.212:41587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:55.196152 11058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:55.196406 11058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:55.201841 11063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:55.202314 11064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:55.202555 11066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:55.203192 11058 server_base.cc:1034] running on GCE node
I20260620 08:05:55.203418 11058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:55.203819 11058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:55.205118 11058 hybrid_clock.cc:648] HybridClock initialized: now 1781942755205085 us; error 54 us; skew 500 ppm
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:55.208544 11058 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:05:55.209159 11058 webserver.cc:460] Webserver started at http://127.31.115.193:34451/ using document root <none> and password file <none>
I20260620 08:05:55.209445 11058 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:55.209522 11058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:55.209739 11058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:05:55.225567 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "21685d3d9abc4d92a53861c2a2dd12eb"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "0618c6db11adb8b9bfa9f6195c8a2237"
server_key_iv: "d3a66babcce3473f56e46a9b7dd8f4e1"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.226235 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "21685d3d9abc4d92a53861c2a2dd12eb"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "0618c6db11adb8b9bfa9f6195c8a2237"
server_key_iv: "d3a66babcce3473f56e46a9b7dd8f4e1"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.228672 11058 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:05:55.249233 11075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.249504 11058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:05:55.249601 11058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "21685d3d9abc4d92a53861c2a2dd12eb"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "0618c6db11adb8b9bfa9f6195c8a2237"
server_key_iv: "d3a66babcce3473f56e46a9b7dd8f4e1"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.249835 11058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:55.273152 11058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:55.275681 11058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:55.275848 11058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:55.276088 11058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:55.276515 11058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:55.276568 11058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.276611 11058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:55.276638 11058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.283383 11058 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:45991
I20260620 08:05:55.283900 11058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:05:55.284463 11188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:45991 every 8 connection(s)
I20260620 08:05:55.291069 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 11058
I20260620 08:05:55.291188 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:05:55.291381 32207 external_mini_cluster.cc:1428] Setting key 2c32ecf13b8792939583dc3376a0081d
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:55.296021 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.284740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:55843 (local address 127.31.115.254:44065)
0620 08:05:55.284918 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.284920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.285166 (+ 246us) server_negotiation.cc:407] Connection header received
0620 08:05:55.285898 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.285900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.285916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.286000 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:55.286806 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.287626 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.289169 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.289533 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.289637 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.292280 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:55.292282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:55.292282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:55.292291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:55.294188 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.294751 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.294753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.294754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.294835 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.295171 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.295172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.295173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.295359 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:55.295440 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.295747 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.295853 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260620 08:05:55.296090 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.284677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44065 (local address 127.31.115.193:55843)
0620 08:05:55.284993 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.285043 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.285779 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.286140 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.286157 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.286412 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:55.286686 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.286692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.287737 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.287738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.289073 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.289076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.289674 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.289675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.289830 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.290494 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:05:55.290505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:55.292142 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:55.294290 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:55.294299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:55.294302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:55.294652 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:55.294923 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:55.294924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:55.294925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:55.295080 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:55.295444 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:55.295452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:55.295604 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.295698 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.295826 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":90,"threads_started":1}
I20260620 08:05:55.297014 11189 heartbeater.cc:347] Connected to a master server at 127.31.115.254:44065
I20260620 08:05:55.297127 11189 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:55.297334 11189 heartbeater.cc:511] Master 127.31.115.254:44065 requested a full tablet report, sending...
I20260620 08:05:55.297991 10977 ts_manager.cc:194] Registered new tserver with Master: 21685d3d9abc4d92a53861c2a2dd12eb (127.31.115.193:45991)
I20260620 08:05:55.298916 10977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:55843
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:05:55.327032 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:41587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44065
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 08:05:55.453456 11195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:55.453801 11195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:55.453845 11195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:55.456046 11195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:55.456192 11195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:55.456282 11195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:05:55.459031 11195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.115.212:41587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:55.459568 11195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:55.459930 11195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:55.467270 11203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:55.467525 11195 server_base.cc:1034] running on GCE node
W20260620 08:05:55.467259 11205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:55.469807 11202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:55.470155 11195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:55.470506 11195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:55.471704 11195 hybrid_clock.cc:648] HybridClock initialized: now 1781942755471679 us; error 36 us; skew 500 ppm
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:55.475449 11195 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:05:55.476084 11195 webserver.cc:460] Webserver started at http://127.31.115.194:45829/ using document root <none> and password file <none>
I20260620 08:05:55.476390 11195 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:55.476454 11195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:55.476642 11195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:05:55.499222 11195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "164f8574f65546079c327df570983f6a"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "f9996b294c1d0380e5926a3d25714bde"
server_key_iv: "4d6373086bd1853d982d4b9728acb404"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.499753 11195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "164f8574f65546079c327df570983f6a"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "f9996b294c1d0380e5926a3d25714bde"
server_key_iv: "4d6373086bd1853d982d4b9728acb404"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.501765 11195 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:05:55.518564 11213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.518841 11195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:05:55.518949 11195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "164f8574f65546079c327df570983f6a"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "f9996b294c1d0380e5926a3d25714bde"
server_key_iv: "4d6373086bd1853d982d4b9728acb404"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.519047 11195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:55.532234 11195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:55.534539 11195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:55.534737 11195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:55.535038 11195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:55.535437 11195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:55.535507 11195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.535560 11195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:55.535596 11195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.543212 11195 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:33419
I20260620 08:05:55.543349 11326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:33419 every 8 connection(s)
I20260620 08:05:55.543773 11195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:05:55.546093 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 11195
I20260620 08:05:55.546204 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:05:55.546424 32207 external_mini_cluster.cc:1428] Setting key d3b34103663729aacfb840170f5b61f4
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:05:55.558816 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.546603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:41589 (local address 127.31.115.254:44065)
0620 08:05:55.546657 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.546659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.546674 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:05:55.547294 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.547295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.547310 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.547351 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:55.547977 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.548484 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.550102 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.550439 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.550514 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.553225 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:55.553228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:55.553229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:55.553239 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:55.555233 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.555851 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.555852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.555853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.555926 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.556230 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.556231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.556232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.556388 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:55.557780 (+ 1392us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.558419 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.558592 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:05:55.559880 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.546275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44065 (local address 127.31.115.194:41589)
0620 08:05:55.546467 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.546510 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.547198 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.547468 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.547473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.547677 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:55.547875 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.547879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.548594 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.548595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.550005 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.550009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.550558 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.550560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.550736 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.551393 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:05:55.551405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:55.553107 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:55.555342 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:55.555350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:55.555353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:55.555764 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:55.556002 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:55.556003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:55.556003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:55.556150 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:55.557817 (+ 1667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:55.557826 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:55.558144 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.558945 (+ 801us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.559436 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
I20260620 08:05:55.561343 11327 heartbeater.cc:347] Connected to a master server at 127.31.115.254:44065
I20260620 08:05:55.561502 11327 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:55.561761 11327 heartbeater.cc:511] Master 127.31.115.254:44065 requested a full tablet report, sending...
I20260620 08:05:55.562206 10977 ts_manager.cc:194] Registered new tserver with Master: 164f8574f65546079c327df570983f6a (127.31.115.194:33419)
I20260620 08:05:55.562723 10977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:41589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:05:55.577265 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:41587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44065
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:05:55.705538 11332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:05:55.705892 11332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:05:55.705937 11332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:05:55.707937 11332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:05:55.708043 11332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:05:55.708094 11332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:05:55.709781 11332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.115.212:41587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:44065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:05:55.710081 11332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:05:55.710346 11332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:05:55.716522 11337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:55.716536 11338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:05:55.716781 11340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:05:55.716845 11332 server_base.cc:1034] running on GCE node
I20260620 08:05:55.717111 11332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:05:55.717376 11332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:05:55.718534 11332 hybrid_clock.cc:648] HybridClock initialized: now 1781942755718507 us; error 39 us; skew 500 ppm
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:05:55.721170 11332 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:05:55.721787 11332 webserver.cc:460] Webserver started at http://127.31.115.195:43277/ using document root <none> and password file <none>
I20260620 08:05:55.722049 11332 fs_manager.cc:359] Metadata directory not provided
I20260620 08:05:55.722096 11332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:05:55.722239 11332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:05:55.735973 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "f6ffecd032fa44a79c728a672c7b6f93"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "1f31570e0fa8e7099e34317f1228ffe1"
server_key_iv: "d7d0312b39b887f7d108abe80fccfc20"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.736779 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ffecd032fa44a79c728a672c7b6f93"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "1f31570e0fa8e7099e34317f1228ffe1"
server_key_iv: "d7d0312b39b887f7d108abe80fccfc20"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.740307 11332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 08:05:55.764314 11347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.764667 11332 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260620 08:05:55.764792 11332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "f6ffecd032fa44a79c728a672c7b6f93"
format_stamp: "Formatted at 2026-06-20 08:05:55 on dist-test-slave-c4w8"
server_key: "1f31570e0fa8e7099e34317f1228ffe1"
server_key_iv: "d7d0312b39b887f7d108abe80fccfc20"
server_key_version: "kuduclusterkey@0"
I20260620 08:05:55.764917 11332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:05:55.794116 11332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:05:55.796967 11332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:05:55.797144 11332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:05:55.797395 11332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:05:55.797823 11332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:05:55.797879 11332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.797920 11332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:05:55.797940 11332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:05:55.805397 11332 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:38803
I20260620 08:05:55.805876 11332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:05:55.806205 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 11332
I20260620 08:05:55.806298 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:05:55.806496 32207 external_mini_cluster.cc:1428] Setting key 351b7d242582cd23b41e1b553802d5cb
I20260620 08:05:55.808831 11460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:38803 every 8 connection(s)
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:05:55.821583 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.809522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:42741 (local address 127.31.115.254:44065)
0620 08:05:55.809569 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.809572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.809949 (+ 377us) server_negotiation.cc:407] Connection header received
0620 08:05:55.810629 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.810631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.810645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.810735 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:55.811513 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.812287 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.813814 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.814134 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.814226 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.816947 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:55.816949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:55.816949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:55.816960 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:55.818941 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.820259 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.820261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.820261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.820340 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.820742 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.820743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.820743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.820911 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:55.820997 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.821373 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.821472 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:05:55.822275 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.809513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:44065 (local address 127.31.115.195:42741)
0620 08:05:55.809804 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.809847 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.810497 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.810816 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.810821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.811077 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:05:55.811379 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.811385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.812413 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.812414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.813716 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.813719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.814250 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.814251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.814402 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.815062 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:05:55.815074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:05:55.816820 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:05:55.819059 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:55.819069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:55.819072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:55.820154 (+ 1082us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:55.820481 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:05:55.820482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:05:55.820482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:05:55.820648 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:05:55.820998 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:05:55.821005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:05:55.821195 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.821327 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.821516 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260620 08:05:55.823122 11461 heartbeater.cc:347] Connected to a master server at 127.31.115.254:44065
I20260620 08:05:55.823285 11461 heartbeater.cc:464] Registering TS with master...
I20260620 08:05:55.823524 11461 heartbeater.cc:511] Master 127.31.115.254:44065 requested a full tablet report, sending...
I20260620 08:05:55.823977 10977 ts_manager.cc:194] Registered new tserver with Master: f6ffecd032fa44a79c728a672c7b6f93 (127.31.115.195:38803)
I20260620 08:05:55.824486 10977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:42741
I20260620 08:05:55.830948 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:05:55.840747 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.832888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.31.115.254:44065)
0620 08:05:55.832937 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.832941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.833083 (+ 142us) server_negotiation.cc:407] Connection header received
0620 08:05:55.833161 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.833162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.833175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.833240 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:05:55.833979 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.834487 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.835262 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.835506 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.835579 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.837351 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:05:55.837353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:05:55.837353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:05:55.837376 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:05:55.839260 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.839679 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.839680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.839680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.839746 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:05:55.840018 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:05:55.840020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:05:55.840020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:05:55.840199 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:05:55.840286 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.840547 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.840640 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:05:55.843303 10977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:05:55.845747 10977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:05:55.854969 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.849397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.31.115.194:33419)
0620 08:05:55.849565 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.849568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.849806 (+ 238us) server_negotiation.cc:407] Connection header received
0620 08:05:55.849905 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.849907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.849923 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.850025 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:55.850222 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.851379 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.853244 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.854431 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.854492 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.854550 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.854815 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.854859 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260620 08:05:55.855578 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.849045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:33419 (local address 127.0.0.1:58696)
0620 08:05:55.849658 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.849668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.849717 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.850036 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.850039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.850042 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:55.850122 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.850125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.851477 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.851478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.853077 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.853080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.854606 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.854607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.854691 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.854705 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.855350 (+ 645us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.855475 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":6,"thread_start_us":68,"threads_started":1}
I20260620 08:05:55.856575 11261 tablet_service.cc:1467] Processing CreateTablet for tablet eb22f223adb94ea1ace2df60b1120dd3 (DEFAULT_TABLE table=test-table [id=b9b35cc3690c4c8f94bd9add8d88e510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:05:55.856979 11261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb22f223adb94ea1ace2df60b1120dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:55.857650 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.849201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45991 (local address 127.0.0.1:44636)
0620 08:05:55.850126 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.850137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.850428 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.850686 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.850687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.850689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:55.850736 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.850738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.853232 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.853233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.854309 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.854312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.857376 (+ 3064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.857377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.857442 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.857458 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.857508 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.857552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":56,"threads_started":1}
I20260620 08:05:55.857949 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.849323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.31.115.193:45991)
0620 08:05:55.849455 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.849461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.850189 (+ 728us) server_negotiation.cc:407] Connection header received
0620 08:05:55.850530 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.850533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.850549 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.850600 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:55.852092 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.853133 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.856495 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.857209 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.857261 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.857312 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.857804 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.857852 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260620 08:05:55.858484 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.848818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:38803 (local address 127.0.0.1:55532)
0620 08:05:55.849172 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.849217 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.849285 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.850811 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.850812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.850813 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:55.850838 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.850839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.854416 (+ 3577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.854418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.855485 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.855488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.858254 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.858256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.858325 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.858330 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.858363 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.858402 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":84,"threads_started":1}
I20260620 08:05:55.858626 11114 tablet_service.cc:1467] Processing CreateTablet for tablet eb22f223adb94ea1ace2df60b1120dd3 (DEFAULT_TABLE table=test-table [id=b9b35cc3690c4c8f94bd9add8d88e510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:05:55.858889 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.848924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.31.115.195:38803)
0620 08:05:55.850283 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.850288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.850297 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:05:55.850333 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.850335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.850350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.850907 (+ 557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:55.850981 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.851988 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.855626 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.856376 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.858661 (+ 2285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.858736 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.858765 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.858810 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":87,"threads_started":1}
I20260620 08:05:55.858994 11114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb22f223adb94ea1ace2df60b1120dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:55.859499 11395 tablet_service.cc:1467] Processing CreateTablet for tablet eb22f223adb94ea1ace2df60b1120dd3 (DEFAULT_TABLE table=test-table [id=b9b35cc3690c4c8f94bd9add8d88e510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:05:55.859824 11395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb22f223adb94ea1ace2df60b1120dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:05:55.862403 11480 tablet_bootstrap.cc:492] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: Bootstrap starting.
I20260620 08:05:55.863610 11480 tablet_bootstrap.cc:654] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:55.863734 11482 tablet_bootstrap.cc:492] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Bootstrap starting.
I20260620 08:05:55.864035 11480 log.cc:824] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:55.864884 11480 tablet_bootstrap.cc:492] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: No bootstrap required, opened a new log
I20260620 08:05:55.864974 11480 ts_tablet_manager.cc:1397] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:05:55.864987 11482 tablet_bootstrap.cc:654] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:55.865401 11482 log.cc:824] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:55.865665 11481 tablet_bootstrap.cc:492] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb: Bootstrap starting.
I20260620 08:05:55.866596 11481 tablet_bootstrap.cc:654] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb: Neither blocks nor log segments found. Creating new log.
I20260620 08:05:55.866987 11481 log.cc:824] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:05:55.867385 11480 raft_consensus.cc:348] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.867583 11480 raft_consensus.cc:374] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:55.867625 11480 raft_consensus.cc:729] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164f8574f65546079c327df570983f6a, State: Initialized, Role: FOLLOWER
I20260620 08:05:55.867835 11482 tablet_bootstrap.cc:492] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: No bootstrap required, opened a new log
I20260620 08:05:55.867895 11482 ts_tablet_manager.cc:1397] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 08:05:55.868363 11480 consensus_queue.cc:260] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.868660 11480 ts_tablet_manager.cc:1428] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 08:05:55.868784 11327 heartbeater.cc:503] Master 127.31.115.254:44065 was elected leader, sending a full tablet report...
I20260620 08:05:55.870517 11482 raft_consensus.cc:348] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.870697 11482 raft_consensus.cc:374] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:55.870769 11482 raft_consensus.cc:729] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ffecd032fa44a79c728a672c7b6f93, State: Initialized, Role: FOLLOWER
I20260620 08:05:55.870899 11482 consensus_queue.cc:260] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.871212 11461 heartbeater.cc:503] Master 127.31.115.254:44065 was elected leader, sending a full tablet report...
I20260620 08:05:55.871212 11482 ts_tablet_manager.cc:1428] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:05:55.872268 11481 tablet_bootstrap.cc:492] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb: No bootstrap required, opened a new log
I20260620 08:05:55.872356 11481 ts_tablet_manager.cc:1397] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 08:05:55.876353 11481 raft_consensus.cc:348] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.876562 11481 raft_consensus.cc:374] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:05:55.876672 11481 raft_consensus.cc:729] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21685d3d9abc4d92a53861c2a2dd12eb, State: Initialized, Role: FOLLOWER
I20260620 08:05:55.876873 11481 consensus_queue.cc:260] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.877364 11189 heartbeater.cc:503] Master 127.31.115.254:44065 was elected leader, sending a full tablet report...
I20260620 08:05:55.877764 11481 ts_tablet_manager.cc:1428] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:05:55.912164 11487 raft_consensus.cc:482] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:05:55.912310 11487 raft_consensus.cc:504] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.912736 11487 leader_election.cc:288] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 164f8574f65546079c327df570983f6a (127.31.115.194:33419), 21685d3d9abc4d92a53861c2a2dd12eb (127.31.115.193:45991)
I20260620 08:05:55.917531 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.913275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:33419 (local address 127.31.115.195:45277)
0620 08:05:55.913334 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.913347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.913417 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.913807 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.913808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.913810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:55.913877 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.913878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.914828 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.914829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.916273 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.916277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.917204 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.917205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.917299 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.917303 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.917390 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.917435 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 08:05:55.918377 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.913124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:45277 (local address 127.31.115.194:33419)
0620 08:05:55.913172 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.913174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.913637 (+ 463us) server_negotiation.cc:407] Connection header received
0620 08:05:55.913675 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.913677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.913690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.913752 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:55.913951 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.914703 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.916386 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.916965 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.917136 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.917154 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.918252 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.918304 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:05:55.918715 11281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb22f223adb94ea1ace2df60b1120dd3" candidate_uuid: "f6ffecd032fa44a79c728a672c7b6f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164f8574f65546079c327df570983f6a" is_pre_election: true
I20260620 08:05:55.918876 11281 raft_consensus.cc:2454] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ffecd032fa44a79c728a672c7b6f93 in term 0.
I20260620 08:05:55.919535 11349 leader_election.cc:302] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 164f8574f65546079c327df570983f6a, f6ffecd032fa44a79c728a672c7b6f93; no voters:
I20260620 08:05:55.919682 11487 raft_consensus.cc:2781] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:05:55.919745 11487 raft_consensus.cc:482] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:05:55.919772 11487 raft_consensus.cc:3037] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:55.920979 11487 raft_consensus.cc:504] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.921303 11281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb22f223adb94ea1ace2df60b1120dd3" candidate_uuid: "f6ffecd032fa44a79c728a672c7b6f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164f8574f65546079c327df570983f6a"
I20260620 08:05:55.921386 11281 raft_consensus.cc:3037] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:55.922363 11281 raft_consensus.cc:2454] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ffecd032fa44a79c728a672c7b6f93 in term 1.
I20260620 08:05:55.922574 11349 leader_election.cc:302] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 164f8574f65546079c327df570983f6a, f6ffecd032fa44a79c728a672c7b6f93; no voters:
I20260620 08:05:55.922832 11487 leader_election.cc:288] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [CANDIDATE]: Term 1 election: Requested vote from peers 164f8574f65546079c327df570983f6a (127.31.115.194:33419), 21685d3d9abc4d92a53861c2a2dd12eb (127.31.115.193:45991)
I20260620 08:05:55.922883 11487 raft_consensus.cc:2781] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:05:55.922925 11487 raft_consensus.cc:686] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 1 LEADER]: Becoming Leader. State: Replica: f6ffecd032fa44a79c728a672c7b6f93, State: Running, Role: LEADER
I20260620 08:05:55.923020 11487 consensus_queue.cc:237] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } }
I20260620 08:05:55.923463 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.913414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45991 (local address 127.31.115.195:58675)
0620 08:05:55.917649 (+ 4235us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.917662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.917710 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.919854 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:05:55.919856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:05:55.919858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:05:55.919902 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.919904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.920941 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.920943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.921970 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:05:55.921974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.923105 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.923127 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:05:55.923188 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.923192 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:05:55.923247 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:05:55.923289 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4223,"thread_start_us":104,"threads_started":1}
I20260620 08:05:55.924170 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.913676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:58675 (local address 127.31.115.193:45991)
0620 08:05:55.913722 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.913724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.917770 (+ 4046us) server_negotiation.cc:407] Connection header received
0620 08:05:55.917834 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.917836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.917847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.917893 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:05:55.920004 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.920835 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.922186 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.922864 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.922941 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.922952 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.923935 (+ 983us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.923996 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:05:55.924360 10977 catalog_manager.cc:5495] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ffecd032fa44a79c728a672c7b6f93 (127.31.115.195). New cstate: current_term: 1 leader_uuid: "f6ffecd032fa44a79c728a672c7b6f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6ffecd032fa44a79c728a672c7b6f93" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 38803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 } health_report { overall_health: UNKNOWN } } }
I20260620 08:05:55.924444 11124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb22f223adb94ea1ace2df60b1120dd3" candidate_uuid: "f6ffecd032fa44a79c728a672c7b6f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" is_pre_election: true
I20260620 08:05:55.924607 11124 raft_consensus.cc:2454] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ffecd032fa44a79c728a672c7b6f93 in term 0.
I20260620 08:05:55.924736 11125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb22f223adb94ea1ace2df60b1120dd3" candidate_uuid: "f6ffecd032fa44a79c728a672c7b6f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21685d3d9abc4d92a53861c2a2dd12eb"
I20260620 08:05:55.924782 11125 raft_consensus.cc:3037] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:05:55.925778 11125 raft_consensus.cc:2454] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ffecd032fa44a79c728a672c7b6f93 in term 1.
I20260620 08:05:55.931639 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.928041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.31.115.195:38803)
0620 08:05:55.928083 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:05:55.928085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:05:55.928093 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:05:55.928211 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:05:55.928212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:05:55.928233 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:05:55.928298 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:05:55.928529 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.929244 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.930424 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:05:55.930740 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:05:55.930825 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:05:55.931039 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:05:55.931186 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:05:55.931265 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:05:55.931494 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 08:05:55.931518 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 08:05:55 dist-test-slave-c4w8 krb5kdc[10500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:05:55.940757 11125 raft_consensus.cc:1261] T eb22f223adb94ea1ace2df60b1120dd3 P 21685d3d9abc4d92a53861c2a2dd12eb [term 1 FOLLOWER]: Refusing update from remote peer f6ffecd032fa44a79c728a672c7b6f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:05:55.941308 11490 consensus_queue.cc:1035] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21685d3d9abc4d92a53861c2a2dd12eb" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:05:55.943851 11281 raft_consensus.cc:1261] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 1 FOLLOWER]: Refusing update from remote peer f6ffecd032fa44a79c728a672c7b6f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:05:55.944242 11487 consensus_queue.cc:1035] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164f8574f65546079c327df570983f6a" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:05:55.945952 11498 mvcc.cc:205] Tried to move back new op lower bound from 7298837528329965568 to 7298837528261640192. Current Snapshot: MvccSnapshot[applied={T|T < 7298837528329965568}]
I20260620 08:05:55.946892 10976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:05:55.947008 10976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:05:55.947269 11493 mvcc.cc:205] Tried to move back new op lower bound from 7298837528329965568 to 7298837528261640192. Current Snapshot: MvccSnapshot[applied={T|T < 7298837528329965568 or (T in {7298837528329965568})}]
I20260620 08:05:55.947743 10976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 701c1ed0659d4422a454e3da0d3463ec: Sending DeleteTablet for 3 replicas of tablet eb22f223adb94ea1ace2df60b1120dd3
I20260620 08:05:55.948097 11114 tablet_service.cc:1514] Processing DeleteTablet for tablet eb22f223adb94ea1ace2df60b1120dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:05:55 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:44636
I20260620 08:05:55.948191 11261 tablet_service.cc:1514] Processing DeleteTablet for tablet eb22f223adb94ea1ace2df60b1120dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:05:55 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:58696
I20260620 08:05:55.948222 11395 tablet_service.cc:1514] Processing DeleteTablet for tablet eb22f223adb94ea1ace2df60b1120dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:05:55 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:55532
I20260620 08:05:55.948499 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 11058
I20260620 08:05:55.953814 11505 tablet_replica.cc:330] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: stopping tablet replica
I20260620 08:05:55.954064 11505 raft_consensus.cc:2229] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:05:55.954267 11505 raft_consensus.cc:2258] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:05:55.955413 11505 ts_tablet_manager.cc:1905] T eb22f223adb94ea1ace2df60b1120dd3 P 164f8574f65546079c327df570983f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:05:55.957226 10962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:45991 (error 108)
I20260620 08:05:55.957731 11506 tablet_replica.cc:330] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: stopping tablet replica
I20260620 08:05:55.958877 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 11195
I20260620 08:05:55.959890 11506 raft_consensus.cc:2229] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:05:55.960130 11506 raft_consensus.cc:2258] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:05:55.960700 11506 ts_tablet_manager.cc:1905] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:05:55.962467 11506 ts_tablet_manager.cc:1918] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:05:55.965706 11506 log.cc:1196] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/eb22f223adb94ea1ace2df60b1120dd3
I20260620 08:05:55.966019 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:05:55.965884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45991 (local address 127.0.0.1:44644)
0620 08:05:55.965945 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.965979 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:45991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 08:05:55.966156 10962 catalog_manager.cc:4584] TS 21685d3d9abc4d92a53861c2a2dd12eb (127.31.115.193:45991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb22f223adb94ea1ace2df60b1120dd3: Network error: Client connection negotiation failed: client connection to 127.31.115.193:45991: connect: Connection refused (error 111)
I20260620 08:05:55.966586 11506 ts_tablet_manager.cc:1939] T eb22f223adb94ea1ace2df60b1120dd3 P f6ffecd032fa44a79c728a672c7b6f93: Deleting consensus metadata
W20260620 08:05:55.966760 11477 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:05:55.966522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:33419 (local address 127.0.0.1:58698)
0620 08:05:55.966558 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 08:05:55.966568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:05:55.966606 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:05:55.966719 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.194:33419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 08:05:55.967221 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 11332
I20260620 08:05:55.967237 10962 catalog_manager.cc:4841] TS f6ffecd032fa44a79c728a672c7b6f93 (127.31.115.195:38803): tablet eb22f223adb94ea1ace2df60b1120dd3 (table test-table [id=b9b35cc3690c4c8f94bd9add8d88e510]) successfully deleted
I20260620 08:05:55.974766 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 10945
I20260620 08:05:55.984570 32207 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 08:05:56.329993 32207 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 08:05:56.330101 32207 mini_ranger.cc:66] Stopping Ranger...
I20260620 08:05:56.675794 32207 mini_ranger.cc:68] Stopped Ranger
2026-06-20 08:05:56.676 UTC [10530] LOG: received smart shutdown request
2026-06-20 08:05:56.680 UTC [10530] LOG: background worker "logical replication launcher" (PID 10539) exited with exit code 1
2026-06-20 08:05:56.680 UTC [10534] LOG: shutting down
2026-06-20 08:05:56.896 UTC [10530] LOG: database system is shut down
2026-06-20T08:05:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54626 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:05:57 dist-test-slave-c4w8 krb5kdc[11516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:05:57 dist-test-slave-c4w8 krb5kdc[11516](info): set up 2 sockets
Jun 20 08:05:57 dist-test-slave-c4w8 krb5kdc[11516](info): commencing operation
krb5kdc: starting...
W20260620 08:05:58.098210 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:05:58 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:05:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/rangeradmin_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/rangeradmin_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/rangerlookup_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/rangerlookup_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
I20260620 08:05:58.285889 32207 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvjmHcA/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/postgres -l logfile start
2026-06-20 08:06:01.392 UTC [11546] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 08:06:01.392 UTC [11546] LOG: listening on IPv4 address "127.31.115.212", port 35623
2026-06-20 08:06:01.397 UTC [11546] LOG: listening on Unix socket "/tmp/.s.PGSQL.35623"
2026-06-20 08:06:01.422 UTC [11549] LOG: database system was shut down at 2026-06-20 08:05:59 UTC
2026-06-20 08:06:01.426 UTC [11546] LOG: database system is ready to accept connections
I20260620 08:06:02.410696 32207 mini_postgres.cc:96] Postgres bound to 35623
2026-06-20 08:06:02.417 UTC [11557] FATAL: database "slave" does not exist
127.31.115.212:35623 - accepting connections
I20260620 08:06:02.417712 32207 mini_ranger.cc:161] Starting Ranger...
I20260620 08:06:02.434404 32207 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 08:06:03.072368 32207 mini_ranger.cc:87] Created ranger Postgres database
I20260620 08:06:03.072463 32207 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-admin
2026-06-20 08:06:03,378 [I] DB FLAVOR :POSTGRES
2026-06-20 08:06:03,379 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:06:03,379 [I] Checking connection..
2026-06-20 08:06:03,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 08:06:03,641 [I] Checking connection passed.
2026-06-20 08:06:03,641 [I] --------- Verifying version history table ---------
2026-06-20 08:06:03,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:06:03,897 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:06:03,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:06:04,149 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:06:04,149 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 08:06:04,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 08:06:04.437 UTC [11621] WARNING: there is no transaction in progress
2026-06-20 08:06:04,444 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 08:06:04,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:06:04,719 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 08:06:04,720 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:06:04,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 08:06:04,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 08:06:05,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c4w8.c.gcp-upstream.internal','N') ;"
2026-06-20 08:06:05,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:06:05,790 [I] Table x_portal_user does not exist in database ranger
2026-06-20 08:06:05,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:06:06,071 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 08:06:06,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:06:06,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:06:06,625 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 08:06:06,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 08:06:07.743 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.762 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.780 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.797 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.815 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.837 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.906 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.914 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.935 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.947 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.960 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.977 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.989 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:07.997 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:08.007 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:08.016 UTC [11729] WARNING: there is no transaction in progress
2026-06-20 08:06:08,879 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 08:06:08,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 08:06:09,148 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 08:06:09,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:06:09,403 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 08:06:09,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:06:09,663 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 08:06:09,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:06:09,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:06:10,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c4w8.c.gcp-upstream.internal';"
2026-06-20 08:06:10,467 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 08:06:10,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 08:06:10,710 [I] DB_PATCHES have already been applied
I20260620 08:06:10.716578 32207 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvjmHcA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 08:06:10.716647 32207 mini_ranger.cc:193] Using host: 127.31.115.212
I20260620 08:06:10.720080 32207 mini_ranger.cc:227] Ranger admin URL: http://127.31.115.212:43779
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:06:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 08:06:11.856473 32207 mini_ranger.cc:160] Time spent starting Ranger: real 9.439s user 0.000s sys 0.009s
Jun 20, 2026 8:06:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:06:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:06:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:06:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:06:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 08:06:36.383208 32207 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.115.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.115.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/rangerkms_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.115.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/rangerkms_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.115.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.115.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.115.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/HTTP_127.31.115.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 08:06:36 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942796, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 08:06:36.490931 32207 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 08:06:36.502075 32207 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 08:06:37.570721 32207 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 08:06:37.570808 32207 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms
I20260620 08:06:37.570827 32207 mini_ranger_kms.cc:227] Using postgres at 127.31.115.212:35623
2026-06-20 08:06:37,740 [I] DB FLAVOR :POSTGRES
2026-06-20 08:06:37,740 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:06:37,740 [I] Checking connection
2026-06-20 08:06:37,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 08:06:38,025 [I] connection success
2026-06-20 08:06:38,026 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 08:06:38,026 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 08:06:38,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 08:06:38,280 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 08:06:38,280 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:06:38,280 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 08:06:38,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.115.212:35623/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 08:06:38,549 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 08:06:39.081188 32207 mini_ranger_kms.cc:310] Created kms service
I20260620 08:06:39.282178 32207 mini_ranger_kms.cc:326] Created kudu user
I20260620 08:06:39.327682 32207 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 08:06:39 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942796, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:39.773674 32207 mini_ranger_kms.cc:384] Added ranger policy
I20260620 08:06:39.773893 32207 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvjmHcA/build/release/bin/postgresql.jar:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvjmHcA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 08:06:39.773918 32207 mini_ranger_kms.cc:240] Using host: 127.31.115.212
I20260620 08:06:39.777204 32207 mini_ranger_kms.cc:276] Ranger KMS PID: 11910
I20260620 08:06:39.777297 32207 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.115.212:48447
Jun 20, 2026 8:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:06:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:06:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 08:06:42.092794 32207 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.602s user 0.019s sys 0.133s
I20260620 08:06:42.092908 32207 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 08:06:42.092971 32207 mini_ranger_kms.cc:401] 127.31.115.212:48447/kms/v1/keys
Jun 20 08:06:42 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781942802, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.115.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 08:06:42 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942802, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.115.212@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
Jun 20 08:06:43 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942802, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.115.212@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
Jun 20 08:06:45 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:06:45.501217 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34271
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:34271
--ranger_config_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:45.589075 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:45.589293 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:45.589318 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:45.590588 11960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:45.590636 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:45.590649 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:45.590662 11960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:45.590674 11960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:45.592299 11960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.115.212:48447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:34271
--ranger_config_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:45.592574 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:45.592823 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:45.598171 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:45.598176 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:45.598316 11960 server_base.cc:1034] running on GCE node
W20260620 08:06:45.598171 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:45.598824 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:45.599155 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:45.600349 11960 hybrid_clock.cc:648] HybridClock initialized: now 1781942805600334 us; error 28 us; skew 500 ppm
Jun 20 08:06:45 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:45.607705 11960 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:45.608310 11960 webserver.cc:460] Webserver started at http://127.31.115.254:34591/ using document root <none> and password file <none>
I20260620 08:06:45.608568 11960 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:45.608629 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:45.608781 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:06:45 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:45.658174 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371"
format_stamp: "Formatted at 2026-06-20 08:06:45 on dist-test-slave-c4w8"
server_key: "462f9030112950f84486de5c021e14a8"
server_key_iv: "79c4cd9206bd8fc1234129a9e9d5e516"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:45.658617 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371"
format_stamp: "Formatted at 2026-06-20 08:06:45 on dist-test-slave-c4w8"
server_key: "462f9030112950f84486de5c021e14a8"
server_key_iv: "79c4cd9206bd8fc1234129a9e9d5e516"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:45.660166 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:06:45.673360 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:45.673647 11960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:06:45.673750 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371"
format_stamp: "Formatted at 2026-06-20 08:06:45 on dist-test-slave-c4w8"
server_key: "462f9030112950f84486de5c021e14a8"
server_key_iv: "79c4cd9206bd8fc1234129a9e9d5e516"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:45.673875 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:45.682739 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:45.684851 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:45.685072 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:45.690240 11960 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:34271
I20260620 08:06:45.690267 12027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:34271 every 8 connection(s)
I20260620 08:06:45.690652 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:45.691414 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:45.693989 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Bootstrap starting.
I20260620 08:06:45.694715 12028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:45.695019 12028 log.cc:824] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:45.695828 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: No bootstrap required, opened a new log
I20260620 08:06:45.697988 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 11960
I20260620 08:06:45.698094 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:06:45.698287 32207 external_mini_cluster.cc:1428] Setting key 6c05ba1a3b037ad26eacf47628343e82
I20260620 08:06:45.698196 12028 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:45.698392 12028 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:45.698482 12028 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9490d6d4fe4a61bc6d0f269e3f1371, State: Initialized, Role: FOLLOWER
I20260620 08:06:45.698693 12028 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:45.698803 12028 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:06:45.698858 12028 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:06:45.698926 12028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:45.699723 12028 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:45.699843 12028 leader_election.cc:302] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9490d6d4fe4a61bc6d0f269e3f1371; no voters:
I20260620 08:06:45.700017 12028 leader_election.cc:288] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:06:45.700071 12033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:45.700258 12033 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 LEADER]: Becoming Leader. State: Replica: ea9490d6d4fe4a61bc6d0f269e3f1371, State: Running, Role: LEADER
I20260620 08:06:45.700371 12028 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:45.700410 12033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:45.700786 12033 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } } }
I20260620 08:06:45.700951 12033 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:45.701159 12033 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9490d6d4fe4a61bc6d0f269e3f1371. Latest consensus state: current_term: 1 leader_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } } }
I20260620 08:06:45.701267 12033 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:45.701431 12042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:45.702106 12042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:06:45.702469 12028 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:06:45.706997 12042 catalog_manager.cc:1347] Generated new cluster ID: ba29f41157bf43558e25bf6ff6687158
I20260620 08:06:45.707180 12042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:06:45 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:45.718441 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:45.698812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.31.115.254:34271)
0620 08:06:45.699026 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:45.699046 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:45.699125 (+ 79us) server_negotiation.cc:407] Connection header received
0620 08:06:45.699266 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:45.699274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:45.699334 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:45.699496 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:45.700271 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:45.701671 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:45.702774 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:45.705562 (+ 2788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:45.705781 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:45.708942 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:45.708948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:45.708948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:45.708961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:45.711057 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:45.711466 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:45.711470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:45.711471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:45.711587 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:45.711789 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:45.711790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:45.711790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:45.712111 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:45.712240 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:45.718018 (+ 5778us) server_negotiation.cc:299] Negotiation successful
0620 08:06:45.718183 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260620 08:06:45.718664 12042 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:06:45.720453 12042 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:06:45.740422 12042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Generated new TSK 0
I20260620 08:06:45.741209 12042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:06:46.534124 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34271
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:46.625530 12073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:46.625874 12073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:46.625916 12073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:46.627125 12073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:46.627197 12073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:46.627243 12073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:06:46.628767 12073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.115.212:48447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:46.629050 12073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:46.629303 12073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:46.634618 12079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:46.634615 12078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:46.634646 12081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:46.635061 12073 server_base.cc:1034] running on GCE node
I20260620 08:06:46.635288 12073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:46.635577 12073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:46.636797 12073 hybrid_clock.cc:648] HybridClock initialized: now 1781942806636775 us; error 34 us; skew 500 ppm
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:46.639585 12073 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:06:46.640147 12073 webserver.cc:460] Webserver started at http://127.31.115.193:37749/ using document root <none> and password file <none>
I20260620 08:06:46.640416 12073 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:46.640462 12073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:46.640619 12073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:46.653412 12073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "0b62b3d9eac74bb38d43cc3e8def391c"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "b6069f4e5e7261286b9f46f2135414e0"
server_key_iv: "078a61c7f664f5047170e28b099d88ff"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:46.654052 12073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "0b62b3d9eac74bb38d43cc3e8def391c"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "b6069f4e5e7261286b9f46f2135414e0"
server_key_iv: "078a61c7f664f5047170e28b099d88ff"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:46.655762 12073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:46.668895 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:46.669147 12073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:46.669250 12073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "0b62b3d9eac74bb38d43cc3e8def391c"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "b6069f4e5e7261286b9f46f2135414e0"
server_key_iv: "078a61c7f664f5047170e28b099d88ff"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:46.669368 12073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:46.681180 12073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:46.684185 12073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:46.684440 12073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:46.684777 12073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:46.685307 12073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:46.685410 12073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:46.685928 12073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:46.686043 12073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:46.695680 12073 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:38293
I20260620 08:06:46.696131 12073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:06:46.701589 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12073
I20260620 08:06:46.701740 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:06:46.701949 32207 external_mini_cluster.cc:1428] Setting key 9c2cb56474584b0241b56cd8397e3eca
I20260620 08:06:46.706146 12201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:38293 every 8 connection(s)
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:46.712988 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:46.697490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:49403 (local address 127.31.115.254:34271)
0620 08:06:46.698412 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:46.698413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:46.698420 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:06:46.698444 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:46.698444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:46.698455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:46.698565 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:46.699395 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.700017 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.705784 (+ 5767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.706123 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.706176 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:46.709085 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:46.709087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:46.709087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:46.709097 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:46.711087 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:46.711742 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:46.711744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:46.711745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:46.711816 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:46.712137 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:46.712138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:46.712138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:46.712298 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:46.712342 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:46.712782 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 08:06:46.712884 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":131,"threads_started":1}
I20260620 08:06:46.713663 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:46.696951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34271 (local address 127.31.115.193:49403)
0620 08:06:46.697721 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:46.697767 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:46.698265 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:46.698651 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:46.698656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:46.698942 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:46.699294 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:46.699302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.700134 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.700135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:46.705645 (+ 5510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:46.705652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.706312 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.706313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:46.706475 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:46.707146 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:46.707157 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:46.708905 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:46.711206 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:46.711215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:46.711218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:46.711639 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:46.711885 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:46.711885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:46.711886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:46.712056 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:46.712392 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:46.712398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:46.712604 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 08:06:46.713176 (+ 572us) client_negotiation.cc:240] Negotiation successful
0620 08:06:46.713362 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":692,"threads_started":1}
I20260620 08:06:46.714316 12202 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34271
I20260620 08:06:46.714457 12202 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:46.714672 12202 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:46.715305 11992 ts_manager.cc:194] Registered new tserver with Master: 0b62b3d9eac74bb38d43cc3e8def391c (127.31.115.193:38293)
I20260620 08:06:46.716243 11992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:49403
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:06:46.755676 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34271
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:46.876734 12210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:46.877038 12210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:46.877075 12210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:46.879016 12210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:46.879086 12210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:46.879129 12210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:06:46.881489 12210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.115.212:48447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:46.881821 12210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:46.882161 12210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:46.889042 12220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:46.889173 12218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:46.889405 12217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:46.889799 12210 server_base.cc:1034] running on GCE node
I20260620 08:06:46.890038 12210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:46.890336 12210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:46.891520 12210 hybrid_clock.cc:648] HybridClock initialized: now 1781942806891487 us; error 46 us; skew 500 ppm
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:46.895015 12210 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:06:46.895669 12210 webserver.cc:460] Webserver started at http://127.31.115.194:43563/ using document root <none> and password file <none>
I20260620 08:06:46.895996 12210 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:46.896075 12210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:46.896266 12210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:46.912762 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "a679c5ad49dd4823b87aa40f55672034"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "749b020c6863b99e23cf1d1dc51adf2f"
server_key_iv: "dc190b36367e0ed85bf097fc4a7881f0"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:46.913455 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "a679c5ad49dd4823b87aa40f55672034"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "749b020c6863b99e23cf1d1dc51adf2f"
server_key_iv: "dc190b36367e0ed85bf097fc4a7881f0"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:46.922250 12210 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260620 08:06:46.950603 12227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:46.950901 12210 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260620 08:06:46.951057 12210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "a679c5ad49dd4823b87aa40f55672034"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "749b020c6863b99e23cf1d1dc51adf2f"
server_key_iv: "dc190b36367e0ed85bf097fc4a7881f0"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:46.951471 12210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:46.963044 12210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:46.965334 12210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:46.965478 12210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:46.965720 12210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:46.966183 12210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:46.966261 12210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:46.966317 12210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:46.966360 12210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:46.974135 12210 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:41949
I20260620 08:06:46.974211 12340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:41949 every 8 connection(s)
I20260620 08:06:46.974599 12210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:06:46.976226 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12210
I20260620 08:06:46.976374 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:06:46.976584 32207 external_mini_cluster.cc:1428] Setting key 5eb12826424993b409e53737ef30f505
Jun 20 08:06:46 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:46.987370 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:46.975503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:33133 (local address 127.31.115.254:34271)
0620 08:06:46.975554 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:46.975556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:46.975800 (+ 244us) server_negotiation.cc:407] Connection header received
0620 08:06:46.976551 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:46.976553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:46.976572 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:46.976659 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:46.977592 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.978397 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.979933 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.980173 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.980240 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:46.983269 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:46.983271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:46.983272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:46.983282 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:46.985294 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:46.985947 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:46.985949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:46.985949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:46.986021 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:46.986485 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:46.986486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:46.986487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:46.986666 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:46.986742 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:46.987141 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 08:06:46.987248 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:06:46.987545 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:46.975486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34271 (local address 127.31.115.194:33133)
0620 08:06:46.975679 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:46.975716 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:46.976414 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:46.976830 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:46.976835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:46.977183 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:46.977476 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:46.977483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.978517 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.978518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:46.979819 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:46.979822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:46.980289 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:46.980290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:46.980445 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:46.981089 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:46.981101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:46.983122 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:46.985432 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:46.985445 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:46.985448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:46.985851 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:46.986106 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:46.986107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:46.986108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:46.986269 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:46.986751 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:46.986759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:46.986961 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 08:06:46.987095 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 08:06:46.987289 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
I20260620 08:06:46.988408 12341 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34271
I20260620 08:06:46.988540 12341 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:46.988744 12341 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:46.989253 11992 ts_manager.cc:194] Registered new tserver with Master: a679c5ad49dd4823b87aa40f55672034 (127.31.115.194:41949)
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:06:46.989765 11992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:33133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:06:47.014274 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34271
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:47.137485 12347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:47.137897 12347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:47.137986 12347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:47.139941 12347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:47.140089 12347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:47.140182 12347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:06:47.142625 12347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.115.212:48447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:47.142933 12347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:47.143292 12347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:47.149950 12352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.150086 12355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.149947 12353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:47.150365 12347 server_base.cc:1034] running on GCE node
I20260620 08:06:47.150730 12347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:47.151072 12347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:47.152355 12347 hybrid_clock.cc:648] HybridClock initialized: now 1781942807152243 us; error 36 us; skew 500 ppm
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:47.156124 12347 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:06:47.156808 12347 webserver.cc:460] Webserver started at http://127.31.115.195:43015/ using document root <none> and password file <none>
I20260620 08:06:47.157138 12347 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:47.157228 12347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:47.157377 12347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:47.172447 12347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "998148fa27944a64b3cc93c4b24c1cb7"
format_stamp: "Formatted at 2026-06-20 08:06:47 on dist-test-slave-c4w8"
server_key: "2df7ea0822662f64c69df7f629b7ec79"
server_key_iv: "daf318a0d96029b7f282369d1622ade5"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:47.172956 12347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "998148fa27944a64b3cc93c4b24c1cb7"
format_stamp: "Formatted at 2026-06-20 08:06:47 on dist-test-slave-c4w8"
server_key: "2df7ea0822662f64c69df7f629b7ec79"
server_key_iv: "daf318a0d96029b7f282369d1622ade5"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:47.181443 12347 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.003s
I20260620 08:06:47.196108 12362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.196727 12347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:06:47.196882 12347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "998148fa27944a64b3cc93c4b24c1cb7"
format_stamp: "Formatted at 2026-06-20 08:06:47 on dist-test-slave-c4w8"
server_key: "2df7ea0822662f64c69df7f629b7ec79"
server_key_iv: "daf318a0d96029b7f282369d1622ade5"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:47.197013 12347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:47.232450 12347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:47.234972 12347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:47.235209 12347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:47.235514 12347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:47.236189 12347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:47.236316 12347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.236402 12347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:47.236500 12347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.248481 12347 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:42433
I20260620 08:06:47.249115 12347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:06:47.249898 12475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:42433 every 8 connection(s)
I20260620 08:06:47.256636 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12347
I20260620 08:06:47.256760 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:06:47.256956 32207 external_mini_cluster.cc:1428] Setting key 07ddc022084c054eecb7dddc039dc653
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:47.266431 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:47.251001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:38895 (local address 127.31.115.254:34271)
0620 08:06:47.251053 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:47.251056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:47.256584 (+ 5528us) server_negotiation.cc:407] Connection header received
0620 08:06:47.257837 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:47.257838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:47.257852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:47.257888 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:47.258608 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.259111 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.260140 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.260343 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.260434 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:47.262696 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:47.262698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:47.262699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:47.262707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:47.264468 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:47.265019 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:47.265021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:47.265021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:47.265088 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:47.265401 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:47.265402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:47.265402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:47.265582 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:47.265741 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:47.266236 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 08:06:47.266334 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:06:47.266791 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:47.250656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34271 (local address 127.31.115.195:38895)
0620 08:06:47.256314 (+ 5658us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:47.256456 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:47.257715 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:47.258100 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:47.258104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:47.258303 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:47.258510 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:47.258514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.259214 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.259215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:47.260040 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:47.260042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.260463 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.260464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:47.260597 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:47.261185 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:47.261193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:47.262583 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:47.264578 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:47.264586 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:47.264590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:47.264908 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:47.265168 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:47.265169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:47.265169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:47.265316 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:47.265795 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:47.265803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:47.266009 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 08:06:47.266206 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 08:06:47.266381 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5594,"thread_start_us":103,"threads_started":1}
I20260620 08:06:47.267295 12476 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34271
I20260620 08:06:47.267424 12476 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.267737 12476 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:47.268280 11992 ts_manager.cc:194] Registered new tserver with Master: 998148fa27944a64b3cc93c4b24c1cb7 (127.31.115.195:42433)
I20260620 08:06:47.268726 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:06:47.268802 11992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:38895
I20260620 08:06:47.269397 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12073
I20260620 08:06:47.276670 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12210
I20260620 08:06:47.282225 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12347
I20260620 08:06:47.288620 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 11960
I20260620 08:06:47.295241 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34271
--webserver_interface=127.31.115.254
--webserver_port=34591
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:34271
--ranger_config_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:47.377696 12479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:47.377930 12479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:47.377955 12479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:47.379208 12479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:47.379259 12479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:47.379274 12479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:47.379287 12479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:47.379299 12479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:47.381006 12479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:34271
--ranger_config_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:47.381276 12479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:47.381541 12479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:47.386453 12484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.386479 12487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.386479 12485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:47.386615 12479 server_base.cc:1034] running on GCE node
I20260620 08:06:47.386976 12479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:47.387346 12479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:47.388571 12479 hybrid_clock.cc:648] HybridClock initialized: now 1781942807388525 us; error 52 us; skew 500 ppm
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:47.391568 12479 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:47.392166 12479 webserver.cc:460] Webserver started at http://127.31.115.254:34591/ using document root <none> and password file <none>
I20260620 08:06:47.392445 12479 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:47.392550 12479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:47.476516 12479 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:47.477718 12495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.478003 12479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:47.478127 12479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371"
format_stamp: "Formatted at 2026-06-20 08:06:45 on dist-test-slave-c4w8"
server_key: "462f9030112950f84486de5c021e14a8"
server_key_iv: "79c4cd9206bd8fc1234129a9e9d5e516"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:47.478637 12479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:47.530629 12479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:47.532543 12479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:47.532716 12479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:47.536985 12479 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:34271
I20260620 08:06:47.537022 12547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:34271 every 8 connection(s)
I20260620 08:06:47.537400 12479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:47.538700 12548 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:06:47.539410 12548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Bootstrap starting.
I20260620 08:06:47.542359 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12479
I20260620 08:06:47.542656 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:38293
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:34271
--webserver_port=37749
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:47.547641 12548 log.cc:824] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:47.549875 12548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:06:47.550181 12548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Bootstrap complete.
I20260620 08:06:47.552627 12548 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:47.552883 12548 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9490d6d4fe4a61bc6d0f269e3f1371, State: Initialized, Role: FOLLOWER
I20260620 08:06:47.553087 12548 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:47.553184 12548 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:06:47.553237 12548 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:06:47.553284 12548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:06:47.554419 12548 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:47.554643 12548 leader_election.cc:302] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9490d6d4fe4a61bc6d0f269e3f1371; no voters:
I20260620 08:06:47.554919 12548 leader_election.cc:288] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:06:47.554965 12552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:06:47.555101 12552 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [term 2 LEADER]: Becoming Leader. State: Replica: ea9490d6d4fe4a61bc6d0f269e3f1371, State: Running, Role: LEADER
I20260620 08:06:47.555265 12548 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:47.555294 12552 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } }
I20260620 08:06:47.555799 12553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9490d6d4fe4a61bc6d0f269e3f1371. Latest consensus state: current_term: 2 leader_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } } }
I20260620 08:06:47.555902 12553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:47.556175 12554 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9490d6d4fe4a61bc6d0f269e3f1371" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34271 } } }
I20260620 08:06:47.556278 12554 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:47.557430 12548 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:06:47.561371 12557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:47.562994 12557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:06:47.563936 12557 catalog_manager.cc:1259] Loaded cluster ID: ba29f41157bf43558e25bf6ff6687158
I20260620 08:06:47.564129 12557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:06:47.567296 12557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:06:47.567662 12557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Loaded TSK: 0
I20260620 08:06:47.567947 12557 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:06:47.667335 12550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:47.667728 12550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:47.667814 12550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:47.669682 12550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:47.669806 12550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:47.669890 12550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:06:47.672276 12550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:38293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=37749
--tserver_master_addrs=127.31.115.254:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:47.672803 12550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:47.673169 12550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:47.679836 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.679941 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.679817 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:47.681211 12550 server_base.cc:1034] running on GCE node
I20260620 08:06:47.681444 12550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:47.681782 12550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:47.682966 12550 hybrid_clock.cc:648] HybridClock initialized: now 1781942807682944 us; error 29 us; skew 500 ppm
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:47.686239 12550 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:06:47.686787 12550 webserver.cc:460] Webserver started at http://127.31.115.193:37749/ using document root <none> and password file <none>
I20260620 08:06:47.687055 12550 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:47.687136 12550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:47.706656 12550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 08:06:47.707755 12596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.707971 12550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:06:47.708053 12550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "0b62b3d9eac74bb38d43cc3e8def391c"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "b6069f4e5e7261286b9f46f2135414e0"
server_key_iv: "078a61c7f664f5047170e28b099d88ff"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:47.708664 12550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:47.724776 12550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:47.727495 12550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:47.727738 12550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:47.728012 12550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:47.728466 12550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:47.728543 12550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.728611 12550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:47.728655 12550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.735565 12550 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:38293
I20260620 08:06:47.736143 12550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:06:47.738555 12709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:38293 every 8 connection(s)
I20260620 08:06:47.742367 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12550
I20260620 08:06:47.742707 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:41949
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:34271
--webserver_port=43563
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:47.752812 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:47.739323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34271 (local address 127.31.115.193:54615)
0620 08:06:47.739525 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:47.739571 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:47.740205 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:47.740761 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:47.740765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:47.740989 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:47.741202 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:47.741208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.742580 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.742581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:47.743916 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:47.743920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.744969 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.744970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:47.745112 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:47.745742 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:47.745755 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:47.747534 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:47.750505 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:47.750516 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:47.750519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:47.750877 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:47.751226 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:47.751227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:47.751227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:47.751386 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:47.752029 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:47.752036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:47.752245 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 08:06:47.752367 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 08:06:47.752558 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
I20260620 08:06:47.752873 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:47.739770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:54615 (local address 127.31.115.254:34271)
0620 08:06:47.739980 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:47.739988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:47.740028 (+ 40us) server_negotiation.cc:407] Connection header received
0620 08:06:47.740468 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:47.740473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:47.740571 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:47.740724 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:47.741339 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.742423 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.744021 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.744730 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.744846 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:47.747687 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:47.747693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:47.747694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:47.747708 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:47.750370 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:47.750991 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:47.750995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:47.750995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:47.751133 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:47.751513 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:47.751514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:47.751515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:47.751926 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:47.752036 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:47.752456 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 08:06:47.752612 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260620 08:06:47.754076 12710 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34271
I20260620 08:06:47.754199 12710 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.754431 12710 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
W20260620 08:06:47.755506 12710 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:34271 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:06:47.755594 12710 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.755681 12710 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:47.756074 12710 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.756184 12710 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
W20260620 08:06:47.756523 12710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:06:47.872491 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:47.872874 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:47.872969 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:47.874857 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:47.875310 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:47.875427 12714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:06:47.877847 12714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:41949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=43563
--tserver_master_addrs=127.31.115.254:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:47.878388 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:47.878728 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:47.885773 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:47.886432 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:47.887858 12714 server_base.cc:1034] running on GCE node
W20260620 08:06:47.893816 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:47.894979 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:47.895336 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:47.902751 12714 hybrid_clock.cc:648] HybridClock initialized: now 1781942807902717 us; error 67 us; skew 500 ppm
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:47.907218 12714 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:06:47.907866 12714 webserver.cc:460] Webserver started at http://127.31.115.194:43563/ using document root <none> and password file <none>
I20260620 08:06:47.908201 12714 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:47.908331 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:47.927244 12714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 08:06:47.928491 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.928720 12714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:47.928810 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "a679c5ad49dd4823b87aa40f55672034"
format_stamp: "Formatted at 2026-06-20 08:06:46 on dist-test-slave-c4w8"
server_key: "749b020c6863b99e23cf1d1dc51adf2f"
server_key_iv: "dc190b36367e0ed85bf097fc4a7881f0"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:47.929409 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:47.940105 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:47.942807 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:47.942976 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:47.943349 12714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:47.943856 12714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:47.943929 12714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.943971 12714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:47.943991 12714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:47.951380 12714 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:41949
I20260620 08:06:47.951985 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:06:47.954205 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12714
I20260620 08:06:47.954548 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.115.212:48447/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:42433
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:34271
--webserver_port=43015
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:47.971306 12843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:41949 every 8 connection(s)
Jun 20 08:06:47 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942807, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:47.995566 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:47.953791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:35575 (local address 127.31.115.254:34271)
0620 08:06:47.953845 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:47.953848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:47.970203 (+ 16355us) server_negotiation.cc:407] Connection header received
0620 08:06:47.970984 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:47.970986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:47.971004 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:47.971084 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:47.979523 (+ 8439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.980321 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.982725 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.983088 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.983137 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:47.987115 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:47.987117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:47.987117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:47.987127 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:47.989033 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:47.994331 (+ 5298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:47.994333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:47.994334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:47.994424 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:47.994766 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:47.994767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:47.994767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:47.994917 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:47.994973 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:47.995322 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 08:06:47.995445 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:06:47.996372 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:47.953220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34271 (local address 127.31.115.194:35575)
0620 08:06:47.969986 (+ 16766us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:47.970073 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:47.970810 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:47.977952 (+ 7142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:47.977960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:47.978669 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:47.979075 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:47.979084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.980456 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.980457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:47.982590 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:47.982595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:47.983264 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:47.983265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:47.983414 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:47.984198 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:47.984211 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:47.986791 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:47.993066 (+ 6275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:47.993080 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:47.993083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:47.993476 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:47.994536 (+ 1060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:47.994537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:47.994537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:47.994691 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:47.995006 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:47.995012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:47.995174 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 08:06:47.995961 (+ 787us) client_negotiation.cc:240] Negotiation successful
0620 08:06:47.996147 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16657,"thread_start_us":765,"threads_started":1}
I20260620 08:06:47.998172 12844 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34271
I20260620 08:06:47.998279 12844 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.998535 12844 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
W20260620 08:06:47.999264 12844 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:34271 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:06:47.999331 12844 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.999403 12844 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:47.999743 12844 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:47.999811 12844 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
W20260620 08:06:48.000108 12844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:06:48.092119 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:48.092437 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:48.092481 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:48.094337 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:48.094434 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:48.094485 12847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:06:48.096925 12847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.115.212:48447/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:42433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=43015
--tserver_master_addrs=127.31.115.254:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:48.097262 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:48.097561 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:48.105146 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:48.105451 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:48.105785 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:48.106101 12847 server_base.cc:1034] running on GCE node
I20260620 08:06:48.106413 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:48.106694 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:48.114082 12847 hybrid_clock.cc:648] HybridClock initialized: now 1781942808114047 us; error 47 us; skew 500 ppm
Jun 20 08:06:48 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:48.117718 12847 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:06:48.118414 12847 webserver.cc:460] Webserver started at http://127.31.115.195:43015/ using document root <none> and password file <none>
I20260620 08:06:48.118754 12847 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:48.118908 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:06:48 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:48.157630 12847 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.001s sys 0.000s
I20260620 08:06:48.168272 12864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:48.168503 12847 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260620 08:06:48.168622 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "998148fa27944a64b3cc93c4b24c1cb7"
format_stamp: "Formatted at 2026-06-20 08:06:47 on dist-test-slave-c4w8"
server_key: "2df7ea0822662f64c69df7f629b7ec79"
server_key_iv: "daf318a0d96029b7f282369d1622ade5"
server_key_version: "kuduclusterkey@0"
I20260620 08:06:48.169126 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:48.194443 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:48.197357 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:48.197523 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:48.197829 12847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:48.198205 12847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:48.198251 12847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:48.198280 12847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:48.198297 12847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:48.206514 12847 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:42433
I20260620 08:06:48.206583 12977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:42433 every 8 connection(s)
I20260620 08:06:48.207050 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:06:48.212106 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 12847
Jun 20 08:06:48 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:48.224941 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:48.212896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.31.115.254:34271)
0620 08:06:48.213050 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:48.213052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:48.213061 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:06:48.213084 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:48.213085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:48.213096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:48.214141 (+ 1045us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:48.214183 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:48.214961 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:48.216158 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:48.216470 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:48.218354 (+ 1884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:48.218417 (+ 63us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:48.218418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:48.218418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:48.218423 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:48.223260 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:48.223700 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:48.223701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:48.223702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:48.223762 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:48.223953 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:48.223954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:48.223954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:48.224085 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:48.224690 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:48.224737 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 08:06:48.224827 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260620 08:06:48.226186 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:48.208342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:59357 (local address 127.31.115.254:34271)
0620 08:06:48.208714 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:48.208717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:48.208795 (+ 78us) server_negotiation.cc:407] Connection header received
0620 08:06:48.209582 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:48.209583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:48.209598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:48.209693 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:48.210486 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:48.211225 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:48.214156 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:48.214488 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:48.214541 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:48.222172 (+ 7631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:48.222174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:48.222175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:48.222185 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:48.224047 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:48.224607 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:48.224609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:48.224609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:48.224670 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:48.224961 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:48.224962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:48.224963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:48.225103 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:48.225145 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:48.225992 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 08:06:48.226097 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260620 08:06:48.226766 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:48.208330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34271 (local address 127.31.115.195:59357)
0620 08:06:48.208649 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:48.208703 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:48.209435 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:48.209789 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:48.209794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:48.210060 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:48.210365 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:48.210373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:48.211320 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:48.211321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:48.214041 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:48.214046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:48.214694 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:48.214695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:48.214856 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:48.218523 (+ 3667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:48.218542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:48.222008 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:48.224149 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:48.224160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:48.224163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:48.224506 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:48.224743 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:48.224744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:48.224744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:48.224883 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:48.225511 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:48.225518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:48.225734 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 08:06:48.226318 (+ 584us) client_negotiation.cc:240] Negotiation successful
0620 08:06:48.226509 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":196,"threads_started":1}
I20260620 08:06:48.227285 12978 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34271
I20260620 08:06:48.227392 12978 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:48.227602 12978 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
W20260620 08:06:48.228358 12978 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:34271 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:06:48.228427 12978 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:48.228493 12978 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:48.228883 12978 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:48.228960 12978 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
W20260620 08:06:48.229275 12978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:06:48.756709 12710 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:48.756889 12710 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:48.757236 12512 ts_manager.cc:194] Registered new tserver with Master: 0b62b3d9eac74bb38d43cc3e8def391c (127.31.115.193:38293)
I20260620 08:06:48.758244 12512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:54615
Jun 20 08:06:48 dist-test-slave-c4w8 krb5kdc[11516](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781942808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:49.000298 12844 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:49.000458 12844 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:49.000821 12512 ts_manager.cc:194] Registered new tserver with Master: a679c5ad49dd4823b87aa40f55672034 (127.31.115.194:41949)
I20260620 08:06:49.001376 12512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:35575
Jun 20 08:06:49 dist-test-slave-c4w8 krb5kdc[11516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942808, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for HTTP/127.31.115.212@KRBTEST.COM
I20260620 08:06:49.229435 12978 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:49.229593 12978 heartbeater.cc:511] Master 127.31.115.254:34271 requested a full tablet report, sending...
I20260620 08:06:49.230063 12512 ts_manager.cc:194] Registered new tserver with Master: 998148fa27944a64b3cc93c4b24c1cb7 (127.31.115.195:42433)
I20260620 08:06:49.230551 12512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:59357
I20260620 08:06:49.234061 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:06:49.246459 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.235100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.31.115.254:34271)
0620 08:06:49.235224 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.235227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.235257 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:06:49.235320 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.235321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.235334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.235423 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:49.236104 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.236881 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.237988 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.239681 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.239766 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.241200 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:49.241202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:49.241202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:49.241211 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:49.244705 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:49.245180 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:49.245181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:49.245181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:49.245244 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:49.245451 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:49.245452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:49.245453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:49.245597 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:49.245712 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.246251 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.246341 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260620 08:06:49.249027 12512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:06:49.250962 12512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:06:49.263988 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.256383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.31.115.193:38293)
0620 08:06:49.256540 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.256545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.259572 (+ 3027us) server_negotiation.cc:407] Connection header received
0620 08:06:49.259681 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.259683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.259699 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.259749 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:49.259969 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.261039 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.262639 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.263472 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.263525 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.263582 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.263858 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.263898 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260620 08:06:49.264232 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.255928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:38293 (local address 127.0.0.1:53408)
0620 08:06:49.259506 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:49.259517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:49.259600 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:49.259847 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:49.259848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:49.259850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:49.259901 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.259902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.261139 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.261140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.262534 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.262537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.263645 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.263646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.263740 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.263754 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:06:49.264111 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 08:06:49.264158 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3560,"thread_start_us":68,"threads_started":1}
I20260620 08:06:49.264547 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.255489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:42433 (local address 127.0.0.1:45026)
0620 08:06:49.257740 (+ 2251us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:49.257752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:49.257807 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:49.258138 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:49.258139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:49.258141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:49.258193 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.258194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.261741 (+ 3547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.261743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.262871 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.262875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.263886 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.263888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.263977 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.263982 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:06:49.264015 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 08:06:49.264476 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"thread_start_us":69,"threads_started":1}
I20260620 08:06:49.265312 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.255608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.31.115.195:42433)
0620 08:06:49.257915 (+ 2307us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.257920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.257929 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:06:49.257963 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.257965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.257980 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.258057 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:49.258290 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.259388 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.262998 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.263774 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.265102 (+ 1328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.265166 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.265192 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.265237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2274,"thread_start_us":57,"threads_started":1}
I20260620 08:06:49.265494 12644 tablet_service.cc:1467] Processing CreateTablet for tablet 99223b1f340f408eac4fdb5ddf150c78 (DEFAULT_TABLE table=test-table [id=da32c543532442c88280dd518b13ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:49.265985 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99223b1f340f408eac4fdb5ddf150c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:49.266184 12909 tablet_service.cc:1467] Processing CreateTablet for tablet 99223b1f340f408eac4fdb5ddf150c78 (DEFAULT_TABLE table=test-table [id=da32c543532442c88280dd518b13ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:49.266501 12909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99223b1f340f408eac4fdb5ddf150c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:49.277849 13012 tablet_bootstrap.cc:492] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Bootstrap starting.
I20260620 08:06:49.278679 13012 tablet_bootstrap.cc:654] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:49.279066 13012 log.cc:824] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:49.279511 13011 tablet_bootstrap.cc:492] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c: Bootstrap starting.
I20260620 08:06:49.280293 13011 tablet_bootstrap.cc:654] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:49.280632 13011 log.cc:824] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:49.282087 13011 tablet_bootstrap.cc:492] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c: No bootstrap required, opened a new log
I20260620 08:06:49.282229 13011 ts_tablet_manager.cc:1397] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:06:49.285789 13011 raft_consensus.cc:348] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.286516 13011 raft_consensus.cc:374] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:49.286657 13011 raft_consensus.cc:729] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b62b3d9eac74bb38d43cc3e8def391c, State: Initialized, Role: FOLLOWER
I20260620 08:06:49.286906 13012 tablet_bootstrap.cc:492] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: No bootstrap required, opened a new log
I20260620 08:06:49.286998 13012 ts_tablet_manager.cc:1397] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 08:06:49.286907 13011 consensus_queue.cc:260] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.287495 13011 ts_tablet_manager.cc:1428] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 08:06:49.287688 12710 heartbeater.cc:503] Master 127.31.115.254:34271 was elected leader, sending a full tablet report...
I20260620 08:06:49.290273 13012 raft_consensus.cc:348] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.290452 13012 raft_consensus.cc:374] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:49.290498 13012 raft_consensus.cc:729] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998148fa27944a64b3cc93c4b24c1cb7, State: Initialized, Role: FOLLOWER
I20260620 08:06:49.290663 13012 consensus_queue.cc:260] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.291021 12978 heartbeater.cc:503] Master 127.31.115.254:34271 was elected leader, sending a full tablet report...
I20260620 08:06:49.291007 13012 ts_tablet_manager.cc:1428] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:06:49.293053 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.255249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41949 (local address 127.0.0.1:54952)
0620 08:06:49.255501 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:49.255601 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:49.255676 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:49.257540 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:49.257544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:49.257546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:49.257658 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.257661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.276030 (+ 18369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.276032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.277236 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.277240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.292732 (+ 15492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.292734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.292831 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.292837 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:06:49.292898 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:06:49.292959 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260620 08:06:49.293658 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.255336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.31.115.194:41949)
0620 08:06:49.255803 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.256662 (+ 859us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.256674 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:06:49.256712 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.256714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.257120 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.257203 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:49.274266 (+ 17063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.275911 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.291611 (+ 15700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.292627 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.292718 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.292775 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.293469 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.293522 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":63,"threads_started":1}
I20260620 08:06:49.294598 12778 tablet_service.cc:1467] Processing CreateTablet for tablet 99223b1f340f408eac4fdb5ddf150c78 (DEFAULT_TABLE table=test-table [id=da32c543532442c88280dd518b13ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:49.294992 12778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99223b1f340f408eac4fdb5ddf150c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:49.302203 13017 tablet_bootstrap.cc:492] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Bootstrap starting.
I20260620 08:06:49.303077 13017 tablet_bootstrap.cc:654] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:49.303505 13017 log.cc:824] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:49.306461 13017 tablet_bootstrap.cc:492] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: No bootstrap required, opened a new log
I20260620 08:06:49.306866 13017 ts_tablet_manager.cc:1397] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:06:49.310148 13017 raft_consensus.cc:348] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.310431 13017 raft_consensus.cc:374] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:49.310531 13017 raft_consensus.cc:729] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a679c5ad49dd4823b87aa40f55672034, State: Initialized, Role: FOLLOWER
I20260620 08:06:49.310763 13017 consensus_queue.cc:260] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.311165 13017 ts_tablet_manager.cc:1428] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:06:49.311342 12844 heartbeater.cc:503] Master 127.31.115.254:34271 was elected leader, sending a full tablet report...
I20260620 08:06:49.346285 13019 raft_consensus.cc:482] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:06:49.346464 13019 raft_consensus.cc:504] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.346894 13019 leader_election.cc:288] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998148fa27944a64b3cc93c4b24c1cb7 (127.31.115.195:42433), 0b62b3d9eac74bb38d43cc3e8def391c (127.31.115.193:38293)
I20260620 08:06:49.351245 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.347330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:58879 (local address 127.31.115.193:38293)
0620 08:06:49.347488 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.347490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.347502 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:06:49.347561 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.347563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.347573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.347639 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:49.347872 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.348617 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.350233 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.350791 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.350842 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.350851 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.351126 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.351170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 08:06:49.351245 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.347423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:50977 (local address 127.31.115.195:42433)
0620 08:06:49.347606 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.347608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.347616 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:06:49.347644 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.347646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.347655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.347705 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:49.347891 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.348622 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.350233 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.350801 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.350842 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.350851 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.351126 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.351171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 08:06:49.351507 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.347071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:42433 (local address 127.31.115.194:50977)
0620 08:06:49.347367 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:49.347375 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:49.347414 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:49.347749 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:49.347750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:49.347751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:49.347814 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.347815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.348729 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.348730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.350121 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.350124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.350940 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.350941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.351044 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.351046 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:06:49.351338 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 08:06:49.351396 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":137,"thread_start_us":110,"threads_started":1}
I20260620 08:06:49.351805 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.347064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:38293 (local address 127.31.115.194:58879)
0620 08:06:49.347324 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:49.347336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:49.347399 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:49.347659 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:49.347661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:49.347663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:49.347761 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.347764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.348729 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.348730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.350121 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:49.350124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.350940 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.350941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:49.351036 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.351041 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:06:49.351683 (+ 642us) client_negotiation.cc:240] Negotiation successful
0620 08:06:49.351721 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":68,"thread_start_us":112,"threads_started":1}
I20260620 08:06:49.351835 12927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99223b1f340f408eac4fdb5ddf150c78" candidate_uuid: "a679c5ad49dd4823b87aa40f55672034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998148fa27944a64b3cc93c4b24c1cb7" is_pre_election: true
I20260620 08:06:49.352042 12927 raft_consensus.cc:2454] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a679c5ad49dd4823b87aa40f55672034 in term 0.
I20260620 08:06:49.352169 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99223b1f340f408eac4fdb5ddf150c78" candidate_uuid: "a679c5ad49dd4823b87aa40f55672034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" is_pre_election: true
I20260620 08:06:49.352306 12733 leader_election.cc:302] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998148fa27944a64b3cc93c4b24c1cb7, a679c5ad49dd4823b87aa40f55672034; no voters:
I20260620 08:06:49.352342 12655 raft_consensus.cc:2454] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a679c5ad49dd4823b87aa40f55672034 in term 0.
I20260620 08:06:49.352453 13019 raft_consensus.cc:2781] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:06:49.352509 13019 raft_consensus.cc:482] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:06:49.352532 13019 raft_consensus.cc:3037] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:49.353448 13019 raft_consensus.cc:504] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.353776 12655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99223b1f340f408eac4fdb5ddf150c78" candidate_uuid: "a679c5ad49dd4823b87aa40f55672034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b62b3d9eac74bb38d43cc3e8def391c"
I20260620 08:06:49.353868 12655 raft_consensus.cc:3037] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:49.354809 12655 raft_consensus.cc:2454] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a679c5ad49dd4823b87aa40f55672034 in term 1.
I20260620 08:06:49.355157 12734 leader_election.cc:302] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b62b3d9eac74bb38d43cc3e8def391c, a679c5ad49dd4823b87aa40f55672034; no voters:
I20260620 08:06:49.355423 12927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99223b1f340f408eac4fdb5ddf150c78" candidate_uuid: "a679c5ad49dd4823b87aa40f55672034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998148fa27944a64b3cc93c4b24c1cb7"
I20260620 08:06:49.355521 12927 raft_consensus.cc:3037] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:49.356431 12927 raft_consensus.cc:2454] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a679c5ad49dd4823b87aa40f55672034 in term 1.
I20260620 08:06:49.356645 13019 leader_election.cc:288] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [CANDIDATE]: Term 1 election: Requested vote from peers 998148fa27944a64b3cc93c4b24c1cb7 (127.31.115.195:42433), 0b62b3d9eac74bb38d43cc3e8def391c (127.31.115.193:38293)
I20260620 08:06:49.356731 13019 raft_consensus.cc:2781] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:49.356777 13019 raft_consensus.cc:686] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 1 LEADER]: Becoming Leader. State: Replica: a679c5ad49dd4823b87aa40f55672034, State: Running, Role: LEADER
I20260620 08:06:49.356868 13019 consensus_queue.cc:237] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } }
I20260620 08:06:49.357945 12512 catalog_manager.cc:5495] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 reported cstate change: term changed from 0 to 1, leader changed from <none> to a679c5ad49dd4823b87aa40f55672034 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "a679c5ad49dd4823b87aa40f55672034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a679c5ad49dd4823b87aa40f55672034" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20260620 08:06:49.366994 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.363784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.31.115.194:41949)
0620 08:06:49.363842 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:49.363844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:49.363877 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:06:49.363971 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:49.363972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:49.363991 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:49.364063 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:06:49.364204 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.364890 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.366058 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:49.366369 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:49.366448 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:49.366621 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:06:49.366757 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:06:49.366845 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:49.366888 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 08:06:49.366912 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:06:49.375101 12655 raft_consensus.cc:1261] T 99223b1f340f408eac4fdb5ddf150c78 P 0b62b3d9eac74bb38d43cc3e8def391c [term 1 FOLLOWER]: Refusing update from remote peer a679c5ad49dd4823b87aa40f55672034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:49.375165 12927 raft_consensus.cc:1261] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 1 FOLLOWER]: Refusing update from remote peer a679c5ad49dd4823b87aa40f55672034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:49.375509 13019 consensus_queue.cc:1035] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b62b3d9eac74bb38d43cc3e8def391c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:49.375644 13022 consensus_queue.cc:1035] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998148fa27944a64b3cc93c4b24c1cb7" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:49.379899 13026 mvcc.cc:205] Tried to move back new op lower bound from 7298837747196665856 to 7298837747126517760. Current Snapshot: MvccSnapshot[applied={T|T < 7298837747196665856}]
I20260620 08:06:49.382200 12511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:06:49.382318 12511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:06:49.383188 12511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9490d6d4fe4a61bc6d0f269e3f1371: Sending DeleteTablet for 3 replicas of tablet 99223b1f340f408eac4fdb5ddf150c78
I20260620 08:06:49.383605 12778 tablet_service.cc:1514] Processing DeleteTablet for tablet 99223b1f340f408eac4fdb5ddf150c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:49 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:54952
I20260620 08:06:49.383890 12909 tablet_service.cc:1514] Processing DeleteTablet for tablet 99223b1f340f408eac4fdb5ddf150c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:49 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:45026
I20260620 08:06:49.384371 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet 99223b1f340f408eac4fdb5ddf150c78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:49 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53408
I20260620 08:06:49.384739 13027 mvcc.cc:205] Tried to move back new op lower bound from 7298837747196665856 to 7298837747126517760. Current Snapshot: MvccSnapshot[applied={T|T < 7298837747196665856 or (T in {7298837747196665856})}]
I20260620 08:06:49.385370 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12550
I20260620 08:06:49.387435 13034 tablet_replica.cc:330] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: stopping tablet replica
I20260620 08:06:49.387598 13034 raft_consensus.cc:2229] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:06:49.387780 13034 raft_consensus.cc:2258] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:49.392359 13035 tablet_replica.cc:330] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: stopping tablet replica
I20260620 08:06:49.392815 13034 ts_tablet_manager.cc:1905] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:06:49.393024 13035 raft_consensus.cc:2229] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:06:49.393205 13035 raft_consensus.cc:2258] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:49.394147 13035 ts_tablet_manager.cc:1905] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:06:49.397121 13034 ts_tablet_manager.cc:1918] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:49.397264 13034 log.cc:1196] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/99223b1f340f408eac4fdb5ddf150c78
I20260620 08:06:49.397657 13034 ts_tablet_manager.cc:1939] T 99223b1f340f408eac4fdb5ddf150c78 P a679c5ad49dd4823b87aa40f55672034: Deleting consensus metadata
I20260620 08:06:49.397739 13035 ts_tablet_manager.cc:1918] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:49.397826 13035 log.cc:1196] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/99223b1f340f408eac4fdb5ddf150c78
I20260620 08:06:49.398128 13035 ts_tablet_manager.cc:1939] T 99223b1f340f408eac4fdb5ddf150c78 P 998148fa27944a64b3cc93c4b24c1cb7: Deleting consensus metadata
I20260620 08:06:49.398418 12498 catalog_manager.cc:4841] TS a679c5ad49dd4823b87aa40f55672034 (127.31.115.194:41949): tablet 99223b1f340f408eac4fdb5ddf150c78 (table test-table [id=da32c543532442c88280dd518b13ab8d]) successfully deleted
I20260620 08:06:49.398533 12498 catalog_manager.cc:4841] TS 998148fa27944a64b3cc93c4b24c1cb7 (127.31.115.195:42433): tablet 99223b1f340f408eac4fdb5ddf150c78 (table test-table [id=da32c543532442c88280dd518b13ab8d]) successfully deleted
W20260620 08:06:49.398849 12499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:38293 (error 108)
I20260620 08:06:49.399710 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:49.399590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:38293 (local address 127.0.0.1:53416)
0620 08:06:49.399644 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:49.399674 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:06:49.399914 12499 catalog_manager.cc:4584] TS 0b62b3d9eac74bb38d43cc3e8def391c (127.31.115.193:38293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99223b1f340f408eac4fdb5ddf150c78: Network error: Client connection negotiation failed: client connection to 127.31.115.193:38293: connect: Connection refused (error 111)
I20260620 08:06:49.400175 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12714
I20260620 08:06:49.407637 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12847
I20260620 08:06:49.414212 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 12479
I20260620 08:06:49.422250 32207 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 08:06:49.767601 32207 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 08:06:49.767725 32207 mini_ranger.cc:66] Stopping Ranger...
I20260620 08:06:50.113752 32207 mini_ranger.cc:68] Stopped Ranger
2026-06-20 08:06:50.114 UTC [11546] LOG: received smart shutdown request
2026-06-20 08:06:50.118 UTC [11546] LOG: background worker "logical replication launcher" (PID 11555) exited with exit code 1
2026-06-20 08:06:50.119 UTC [11550] LOG: shutting down
2026-06-20 08:06:50.345 UTC [11546] LOG: database system is shut down
2026-06-20T08:06:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53460 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:06:50 dist-test-slave-c4w8 krb5kdc[13046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:06:50 dist-test-slave-c4w8 krb5kdc[13046](info): set up 2 sockets
Jun 20 08:06:50 dist-test-slave-c4w8 krb5kdc[13046](info): commencing operation
krb5kdc: starting...
W20260620 08:06:51.541935 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:06:51 dist-test-slave-c4w8 krb5kdc[13046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:06:51.674085 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41333
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41333,127.31.115.253:42245,127.31.115.252:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:51.760780 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:51.761049 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:51.761076 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:51.762290 13059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:51.762341 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:51.762355 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:51.762368 13059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:51.762380 13059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:51.763983 13059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41333,127.31.115.253:42245,127.31.115.252:42297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:41333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:51.764366 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:51.764568 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:51.769917 13067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:51.770040 13059 server_base.cc:1034] running on GCE node
W20260620 08:06:51.769941 13064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:51.769934 13065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:51.770612 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:51.770949 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:51.772151 13059 hybrid_clock.cc:648] HybridClock initialized: now 1781942811772139 us; error 35 us; skew 500 ppm
Jun 20 08:06:51 dist-test-slave-c4w8 krb5kdc[13046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:51.778335 13059 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:51.778882 13059 webserver.cc:460] Webserver started at http://127.31.115.254:36065/ using document root <none> and password file <none>
I20260620 08:06:51.779155 13059 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:51.779227 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:51.779383 13059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:51.780526 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "de772e1c98c741e39bc06dd11dcbf8b4"
format_stamp: "Formatted at 2026-06-20 08:06:51 on dist-test-slave-c4w8"
server_key: "8c5b24d82a5910b76c897911b4ead7cf"
server_key_iv: "aec20729fd50a256d2d1a709561f1be4"
server_key_version: "encryptionkey@0"
I20260620 08:06:51.780900 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "de772e1c98c741e39bc06dd11dcbf8b4"
format_stamp: "Formatted at 2026-06-20 08:06:51 on dist-test-slave-c4w8"
server_key: "8c5b24d82a5910b76c897911b4ead7cf"
server_key_iv: "aec20729fd50a256d2d1a709561f1be4"
server_key_version: "encryptionkey@0"
I20260620 08:06:51.782404 13059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:06:51.783452 13074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:51.783679 13059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:51.783798 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "de772e1c98c741e39bc06dd11dcbf8b4"
format_stamp: "Formatted at 2026-06-20 08:06:51 on dist-test-slave-c4w8"
server_key: "8c5b24d82a5910b76c897911b4ead7cf"
server_key_iv: "aec20729fd50a256d2d1a709561f1be4"
server_key_version: "encryptionkey@0"
I20260620 08:06:51.783885 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:51.797201 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:51.799577 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:51.799768 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:51.804021 13059 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:41333
I20260620 08:06:51.804049 13126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:41333 every 8 connection(s)
I20260620 08:06:51.804415 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:51.805296 13127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:51.806910 13127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:51.810168 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13059
I20260620 08:06:51.810286 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:06:51.810546 32207 external_mini_cluster.cc:1428] Setting key a6710ef200733a9d46a3533b9ec0fde5
Jun 20 08:06:51 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:51.818435 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.807652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41333 (local address 127.0.0.1:59158)
0620 08:06:51.807825 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.807883 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:51.808332 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:51.808709 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:51.808712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:51.809032 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:51.809200 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:51.809205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:51.810384 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:51.810385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:51.811147 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:51.811150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:51.811732 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:51.811733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:51.811808 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:51.812820 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:51.812829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:51.814727 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:51.816760 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:51.816767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:51.816769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:51.816987 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:51.817250 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:51.817251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:51.817251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:51.817349 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:51.817798 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:51.817804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:51.817872 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 08:06:51.817990 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 08:06:51.818237 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":109,"thread_start_us":93,"threads_started":1}
I20260620 08:06:51.818441 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.807711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.31.115.254:41333)
0620 08:06:51.807929 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:51.807935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:51.807978 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:06:51.808539 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:51.808541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:51.808561 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:51.808651 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:51.809324 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:51.810228 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:51.811249 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:51.811577 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:51.811775 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:51.814852 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:51.814860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:51.814861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:51.814869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:51.816640 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:51.817088 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:51.817092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:51.817092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:51.817179 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:51.817424 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:51.817426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:51.817426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:51.817727 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:51.817823 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:51.818017 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 08:06:51.818178 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":171,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.31.115.253@KRBTEST.COM; defaulting to no policy
I20260620 08:06:51.819276 13127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:51.819751 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.819608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46328)
0620 08:06:51.819668 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.819723 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:06:51.819908 13075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
I20260620 08:06:51.820464 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.820297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46334)
0620 08:06:51.820392 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.820423 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 08:06:51.820626 13127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:42245: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab.
Entry for principal kudu/127.31.115.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab.
I20260620 08:06:51.852178 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:42245
--webserver_interface=127.31.115.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab
--principal=kudu/127.31.115.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41333,127.31.115.253:42245,127.31.115.252:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:51.869903 13127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } attempt: 1
I20260620 08:06:51.871419 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.871134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46336)
0620 08:06:51.871278 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.871350 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 08:06:51.871910 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.871674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46348)
0620 08:06:51.871849 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.871876 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
W20260620 08:06:51.872325 13127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:42245: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
I20260620 08:06:51.940722 13127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } attempt: 2
I20260620 08:06:51.941331 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.941168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46350)
0620 08:06:51.941250 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.941296 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 08:06:51.941852 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:51.941725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46352)
0620 08:06:51.941784 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:51.941822 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 08:06:51.942134 13127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:42245: Network error: Client connection negotiation failed: client connection to 127.31.115.253:42245: connect: Connection refused (error 111)
W20260620 08:06:51.950744 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:51.950987 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:51.951013 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:51.952723 13134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:51.952801 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:51.952904 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:51.952967 13134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:51.952986 13134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:51.955760 13134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41333,127.31.115.253:42245,127.31.115.252:42297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab
--principal=kudu/127.31.115.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:42245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.115.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:51.956133 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:51.956468 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:51.962189 13143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:51.962134 13140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:51.962404 13134 server_base.cc:1034] running on GCE node
W20260620 08:06:51.962400 13141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:51.962950 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:51.963296 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:51.964565 13134 hybrid_clock.cc:648] HybridClock initialized: now 1781942811964541 us; error 70 us; skew 500 ppm
Jun 20 08:06:51 dist-test-slave-c4w8 krb5kdc[13046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:51.967566 13134 init.cc:377] Logged in from keytab as kudu/127.31.115.253@KRBTEST.COM (short username kudu)
I20260620 08:06:51.968230 13134 webserver.cc:460] Webserver started at http://127.31.115.253:36535/ using document root <none> and password file <none>
I20260620 08:06:51.968514 13134 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:51.968595 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:51.968765 13134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:51.970065 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data/instance:
uuid: "98ec0bc16c24461290d1c2cafe19577a"
format_stamp: "Formatted at 2026-06-20 08:06:51 on dist-test-slave-c4w8"
server_key: "c3406f30c9d9c5186fdba7e9de4d0ef6"
server_key_iv: "bcc73c2d89c1ad5945b82da1909e3d80"
server_key_version: "encryptionkey@0"
I20260620 08:06:51.970469 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal/instance:
uuid: "98ec0bc16c24461290d1c2cafe19577a"
format_stamp: "Formatted at 2026-06-20 08:06:51 on dist-test-slave-c4w8"
server_key: "c3406f30c9d9c5186fdba7e9de4d0ef6"
server_key_iv: "bcc73c2d89c1ad5945b82da1909e3d80"
server_key_version: "encryptionkey@0"
I20260620 08:06:51.972059 13134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:06:51.973048 13150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:51.973332 13134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:51.973502 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal
uuid: "98ec0bc16c24461290d1c2cafe19577a"
format_stamp: "Formatted at 2026-06-20 08:06:51 on dist-test-slave-c4w8"
server_key: "c3406f30c9d9c5186fdba7e9de4d0ef6"
server_key_iv: "bcc73c2d89c1ad5945b82da1909e3d80"
server_key_version: "encryptionkey@0"
I20260620 08:06:51.973606 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:51.988588 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:51.991354 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:51.991528 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:51.995569 13134 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.253:42245
I20260620 08:06:51.995687 13202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.253:42245 every 8 connection(s)
I20260620 08:06:51.996045 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
I20260620 08:06:51.997099 13203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:51.999383 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13134
I20260620 08:06:51.999115 13203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:51.999544 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-1/wal/instance
I20260620 08:06:51.999873 32207 external_mini_cluster.cc:1428] Setting key e96a451ae3f3ef3245f18dc3f46724dc
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.252@KRBTEST.COM; defaulting to no policy
I20260620 08:06:52.011879 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.000270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.31.115.254:41333)
0620 08:06:52.000383 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.000386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.000613 (+ 227us) server_negotiation.cc:407] Connection header received
0620 08:06:52.001438 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.001440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.001456 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.001522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.002402 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.003273 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.005019 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.005242 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.005309 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.008238 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.008240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.008240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.008251 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.010111 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.010686 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.010688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.010688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.010764 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.011050 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.011051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.011051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.011173 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.011227 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.011702 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.011808 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:06:52.012079 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.000229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41333 (local address 127.0.0.1:59172)
0620 08:06:52.000432 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.000489 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.001312 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.001649 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.001656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.002004 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.002218 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.002223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.003412 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.003413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.004896 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.004901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.005349 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.005351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.005542 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.006238 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.006250 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.008115 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.010215 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.010225 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.010227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.010579 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.010855 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.010856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.010857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.011001 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.011262 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.011269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.011415 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.011525 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.011758 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.252@KRBTEST.COM" created.
I20260620 08:06:52.012832 13203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:06:52.024030 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.013296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46368)
0620 08:06:52.013457 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.013472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.013527 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.013899 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.013900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.014176 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.014259 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.014262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.015544 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.015545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.016401 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.016404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.017206 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.017208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.017276 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.018726 (+ 1450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.018735 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.020334 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.022749 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.022750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.022751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.022967 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.023228 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.023228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.023229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.023288 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.023693 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.023694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.023741 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.023834 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.023954 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 08:06:52.024117 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.013365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.31.115.253:42245)
0620 08:06:52.013681 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.013686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.013718 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:06:52.013745 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.013746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.013758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.013837 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.014434 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.015441 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.016590 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.017105 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.017190 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.020449 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.020473 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.020474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.020482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.022633 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.023079 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.023084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.023084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.023154 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.023356 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.023357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.023357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.023591 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.023691 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.023876 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.024038 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260620 08:06:52.024924 13203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:52.025396 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.025224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41550)
0620 08:06:52.025311 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.025368 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 08:06:52.025655 13154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
I20260620 08:06:52.026326 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.026206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41556)
0620 08:06:52.026273 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.026304 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 08:06:52.026671 13203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:42297: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
I20260620 08:06:52.033363 13127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab.
Entry for principal kudu/127.31.115.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab.
I20260620 08:06:52.035030 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:42297
--webserver_interface=127.31.115.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab
--principal=kudu/127.31.115.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:41333,127.31.115.253:42245,127.31.115.252:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:06:52.043179 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.033927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46382)
0620 08:06:52.034035 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.034047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.034104 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.034261 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.034262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.034523 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.034591 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.034594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.035383 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.035385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.035925 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.035927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.036387 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.036388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.036505 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.037379 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.037387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.039270 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.041748 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.041749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.041750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.042046 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.042323 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.042324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.042324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.042394 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.042743 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.042745 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.042797 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.042964 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.043103 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 08:06:52.043259 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.034014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.31.115.253:42245)
0620 08:06:52.034086 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.034089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.034101 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:06:52.034174 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.034175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.034185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.034255 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.034680 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.035290 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.036048 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.036306 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.036389 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.039471 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.039473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.039474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.039484 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.041579 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.042140 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.042141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.042142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.042217 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.042469 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.042470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.042471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.042644 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.042727 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.042996 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.043115 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:06:52.043854 13127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:52.044245 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.044139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41560)
0620 08:06:52.044189 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.044225 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 08:06:52.044690 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.044600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41562)
0620 08:06:52.044646 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.044673 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:06:52.044797 13127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:42297: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
I20260620 08:06:52.075971 13127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } attempt: 1
I20260620 08:06:52.075965 13203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } attempt: 1
I20260620 08:06:52.076519 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.076360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41574)
0620 08:06:52.076441 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.076481 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 08:06:52.076545 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.076351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41582)
0620 08:06:52.076431 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.076479 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 08:06:52.076936 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.076804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41600)
0620 08:06:52.076859 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.076889 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 08:06:52.077064 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.076791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41590)
0620 08:06:52.077016 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.077041 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 08:06:52.077046 13203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:42297: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
W20260620 08:06:52.077165 13127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:42297: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
W20260620 08:06:52.127738 13210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:52.128003 13210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:52.128032 13210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:52.129418 13210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:52.129478 13210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:52.129496 13210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:52.129511 13210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:52.129525 13210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:52.131438 13210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:41333,127.31.115.253:42245,127.31.115.252:42297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab
--principal=kudu/127.31.115.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.115.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:52.131747 13210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:52.132025 13210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:52.137765 13216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:52.137856 13210 server_base.cc:1034] running on GCE node
W20260620 08:06:52.137794 13218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:52.138001 13215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:52.138288 13210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:52.138868 13210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:52.140110 13210 hybrid_clock.cc:648] HybridClock initialized: now 1781942812140073 us; error 47 us; skew 500 ppm
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:52.143065 13210 init.cc:377] Logged in from keytab as kudu/127.31.115.252@KRBTEST.COM (short username kudu)
I20260620 08:06:52.143623 13210 webserver.cc:460] Webserver started at http://127.31.115.252:39561/ using document root <none> and password file <none>
I20260620 08:06:52.143865 13210 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:52.143939 13210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:52.144098 13210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:52.145308 13203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } attempt: 2
I20260620 08:06:52.145288 13210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data/instance:
uuid: "5f545e67a89147469b0b16b259ed1666"
format_stamp: "Formatted at 2026-06-20 08:06:52 on dist-test-slave-c4w8"
server_key: "2e945468159c4631cf7b2d6774697c27"
server_key_iv: "d7e923ef73d5fea0a5a3c9cf4f9f5882"
server_key_version: "encryptionkey@0"
I20260620 08:06:52.145761 13210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal/instance:
uuid: "5f545e67a89147469b0b16b259ed1666"
format_stamp: "Formatted at 2026-06-20 08:06:52 on dist-test-slave-c4w8"
server_key: "2e945468159c4631cf7b2d6774697c27"
server_key_iv: "d7e923ef73d5fea0a5a3c9cf4f9f5882"
server_key_version: "encryptionkey@0"
I20260620 08:06:52.145891 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.145713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41614)
0620 08:06:52.145779 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.145843 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:06:52.146242 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.146159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41626)
0620 08:06:52.146203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.146227 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:06:52.146354 13203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:42297: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
I20260620 08:06:52.147372 13210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:52.148207 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:52.148432 13210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:52.148524 13210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal
uuid: "5f545e67a89147469b0b16b259ed1666"
format_stamp: "Formatted at 2026-06-20 08:06:52 on dist-test-slave-c4w8"
server_key: "2e945468159c4631cf7b2d6774697c27"
server_key_iv: "d7e923ef73d5fea0a5a3c9cf4f9f5882"
server_key_version: "encryptionkey@0"
I20260620 08:06:52.148603 13210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:52.152309 13127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } attempt: 2
I20260620 08:06:52.152772 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.152638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41634)
0620 08:06:52.152702 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.152744 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:06:52.153173 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.153066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41646)
0620 08:06:52.153120 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.153155 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:06:52.153265 13127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:42297: Network error: Client connection negotiation failed: client connection to 127.31.115.252:42297: connect: Connection refused (error 111)
I20260620 08:06:52.158825 13210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:52.161458 13210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:52.161713 13210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:52.166596 13210 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.252:42297
I20260620 08:06:52.167062 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.252:42297 every 8 connection(s)
I20260620 08:06:52.167088 13210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
I20260620 08:06:52.167902 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:52.169549 13278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:52.171928 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13210
I20260620 08:06:52.172068 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781942645567166-32207-0/minicluster-data/master-2/wal/instance
I20260620 08:06:52.172281 32207 external_mini_cluster.cc:1428] Setting key 04be7e423fb66c1be551074d5e43560d
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:06:52.182068 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.170463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.31.115.254:41333)
0620 08:06:52.170578 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.170581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.170783 (+ 202us) server_negotiation.cc:407] Connection header received
0620 08:06:52.171412 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.171414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.171429 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.171510 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.172441 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.173165 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.174846 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.175212 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.175285 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.178184 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.178186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.178187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.178198 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.180028 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.180624 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.180626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.180626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.180713 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.181174 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.181176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.181176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.181354 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.181456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.181823 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.181931 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 08:06:52.182184 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.170370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:41333 (local address 127.0.0.1:59186)
0620 08:06:52.170618 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.170673 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.171243 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.171611 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.171615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.171987 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.172293 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.172299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.173302 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.173304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.174748 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.174751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.175354 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.175355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.175531 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.176126 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.176137 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.178075 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.180138 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.180147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.180150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.180526 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.180807 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.180808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.180808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.181016 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.181452 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.181459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.181598 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.181765 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.181966 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":116,"threads_started":1}
I20260620 08:06:52.182919 13278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:52.184087 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.172795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.31.115.252:42297)
0620 08:06:52.173101 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.173106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.173130 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:06:52.173152 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.173154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.173175 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.173253 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.174315 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.175198 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.176414 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.176705 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.176772 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.179770 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.179781 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.179781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.179791 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.182243 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.182849 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.182853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.182853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.182923 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.183212 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.183213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.183213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.183425 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.183586 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.183870 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.183987 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":224,"threads_started":1}
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:52.191890 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.183250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:42245 (local address 127.0.0.1:46386)
0620 08:06:52.183305 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.183317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.183360 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.183551 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.183553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.183805 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.183882 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.183884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.185191 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.185193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.185865 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.185867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.186355 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.186356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.186475 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.186982 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.186988 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.188243 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.190038 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.190040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.190040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.190622 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.190861 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.190862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.190862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.190976 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.191549 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.191550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.191608 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.191702 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.191815 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 08:06:52.192055 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.183320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.31.115.253:42245)
0620 08:06:52.183369 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.183371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.183380 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:06:52.183407 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.183408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.183430 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.183509 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.183988 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.184975 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.186032 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.186256 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.186351 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.188346 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.188347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.188347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.188356 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.189936 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.190722 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.190724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.190724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.190783 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.191218 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.191219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.191220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.191414 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.191525 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.191847 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.191946 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:06:52.192524 13278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:06:52.194804 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.185664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.31.115.254:41333)
0620 08:06:52.185725 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.185727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.185734 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:06:52.185752 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.185753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.185761 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.185826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.186697 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.187361 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.188448 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.188720 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.188878 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.191825 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.191827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.191828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.191836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.193469 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.193902 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.193903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.193904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.193977 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.194205 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.194206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.194207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.194338 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.194563 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.194601 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.194696 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942812, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:06:52.201352 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.192917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.31.115.252:42297)
0620 08:06:52.192956 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.192957 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.193645 (+ 688us) server_negotiation.cc:407] Connection header received
0620 08:06:52.193708 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.193709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.193723 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.193794 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.194024 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.194558 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.195459 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.195754 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.195823 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.198086 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.198088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.198088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.198096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.199833 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.200225 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.200226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.200227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.200289 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.200678 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.200679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.200679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.200799 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.200918 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.201078 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.201259 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":20}
I20260620 08:06:52.201375 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.192892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41672)
0620 08:06:52.193554 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.193563 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.193600 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.193796 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.193796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.193912 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.193951 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.193952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.194655 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.194656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.195377 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.195384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.195851 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.195852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.195931 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.196328 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.196335 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.197969 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.199954 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.199955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.199956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.200137 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.200472 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.200473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.200474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.200546 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.200909 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.200910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.200954 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.201054 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.201187 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":3}
I20260620 08:06:52.203011 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: Bootstrap starting.
I20260620 08:06:52.204048 13278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:52.204126 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.195648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.31.115.253:42245)
0620 08:06:52.195690 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.195692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.195700 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:06:52.195751 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.195752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.195767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.195809 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.196763 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.197222 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.198087 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.198290 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.198375 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.200998 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.200999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.201000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.201008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.202882 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.203310 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.203311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.203311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.203374 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.203578 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.203579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.203580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.203700 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.203899 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.203937 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.204013 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:06:52.204496 13278 log.cc:824] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:52.205503 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: No bootstrap required, opened a new log
I20260620 08:06:52.205855 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:06:52.207685 13278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.207875 13278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:52.207924 13278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f545e67a89147469b0b16b259ed1666, State: Initialized, Role: FOLLOWER
I20260620 08:06:52.208096 13278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.208487 13278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:52.210989 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.211074 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:06:52.210731 13302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:06:52.211361 13302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:06:52.211568 13210 master_runner.cc:186] Error getting master registration for 127.31.115.254:41333: OK, instance_id { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" instance_seqno: 1781942811802756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:06:52.214221 13210 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:06:52.216403 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.207213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.31.115.252:42297)
0620 08:06:52.207258 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.207260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.207712 (+ 452us) server_negotiation.cc:407] Connection header received
0620 08:06:52.207740 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.207741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.207753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.207805 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.208774 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.209654 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.210879 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.211195 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.211266 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.212906 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.212907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.212908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.212915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.214753 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.215223 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.215224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.215225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.215289 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.215529 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.215530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.215531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.215715 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.215817 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.216192 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.216287 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:06:52.217970 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.207756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.31.115.253:42245)
0620 08:06:52.207801 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.207803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.207810 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:06:52.207831 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.207832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.207842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.207918 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.208779 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.209498 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.210625 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.210939 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.211010 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.213065 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.213066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.213067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.213074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.215668 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.216354 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.216356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.216356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.216419 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.217141 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.217143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.217143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.217283 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.217400 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.217678 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.217868 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:06:52.218883 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.207244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.31.115.254:41333)
0620 08:06:52.207294 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.207296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.207787 (+ 491us) server_negotiation.cc:407] Connection header received
0620 08:06:52.207883 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.207884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.207895 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.207925 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.208744 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.209508 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.210631 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.211264 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.211308 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.215506 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.215508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.215509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.215517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.217412 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.217910 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.217911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.217912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.217985 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.218247 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.218248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.218249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.218393 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.218488 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.218749 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.218816 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:06:52.237522 13203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } attempt: 3
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:06:52.245121 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.237905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41690)
0620 08:06:52.238016 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.238029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.238079 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.238223 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.238224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.238397 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.238444 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.238446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.239100 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.239101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.239621 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.239623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.239977 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.239978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.240040 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.240517 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.240523 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.242137 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.244062 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.244063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.244063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.244239 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.244449 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.244450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.244451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.244518 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.244779 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.244780 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.244829 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.244917 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.245020 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 08:06:52.245121 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.237987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.31.115.252:42297)
0620 08:06:52.238056 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.238059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.238069 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:06:52.238135 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.238136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.238146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.238219 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.238524 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.239015 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.239706 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.239899 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.239976 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.242238 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.242239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.242240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.242247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.243971 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.244312 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.244313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.244313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.244370 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.244581 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.244582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.244582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.244712 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.244788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.244944 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.245020 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:06:52.246587 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a: Bootstrap starting.
I20260620 08:06:52.247244 13203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:52.247560 13203 log.cc:824] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:52.248333 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a: No bootstrap required, opened a new log
I20260620 08:06:52.249913 13203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.250023 13203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:52.250046 13203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ec0bc16c24461290d1c2cafe19577a, State: Initialized, Role: FOLLOWER
I20260620 08:06:52.250186 13203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.250445 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.250523 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:06:52.250522 13203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:52.252410 13127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } attempt: 3
W20260620 08:06:52.252580 13316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:06:52.252636 13316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:06:52.252827 13134 master_runner.cc:186] Error getting master registration for 127.31.115.254:41333: OK, instance_id { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" instance_seqno: 1781942811802756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:06:52.256814 13134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 08:06:52 dist-test-slave-c4w8 krb5kdc[13046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942811, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:06:52.261133 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.252835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.31.115.252:42297)
0620 08:06:52.252878 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:52.252880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:52.252890 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:06:52.252971 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:52.252973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:52.252986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:52.253034 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:52.253428 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.254316 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.255491 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.255819 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.255885 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.258207 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:52.258208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:52.258209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:52.258218 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:52.259795 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.260304 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.260306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.260306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.260371 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:52.260578 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:52.260579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:52.260580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:52.260712 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:52.260791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:52.260951 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:06:52.261031 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:06:52.261133 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:52.252675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:42297 (local address 127.0.0.1:41694)
0620 08:06:52.252830 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:52.252843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:52.252887 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:52.253077 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:52.253078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:52.253282 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:52.253331 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.253332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.254429 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.254430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.255387 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:52.255390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:52.255947 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:52.255949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:52.256012 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:52.256569 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:52.256576 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:52.258096 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:52.260017 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.260019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.260020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.260226 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.260443 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:52.260444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:52.260445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:52.260509 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:52.260781 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:52.260782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:52.260832 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 08:06:52.260919 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 08:06:52.261031 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 08:06:52.263249 13127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: Bootstrap starting.
I20260620 08:06:52.263906 13127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:52.264217 13127 log.cc:824] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:52.264963 13127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: No bootstrap required, opened a new log
I20260620 08:06:52.266459 13127 raft_consensus.cc:348] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.266595 13127 raft_consensus.cc:374] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:52.266641 13127 raft_consensus.cc:729] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de772e1c98c741e39bc06dd11dcbf8b4, State: Initialized, Role: FOLLOWER
I20260620 08:06:52.266781 13127 consensus_queue.cc:260] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.267074 13319 sys_catalog.cc:455] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.267148 13127 sys_catalog.cc:564] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:52.267213 13319 sys_catalog.cc:458] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:06:52.268965 13330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:06:52.269066 13330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:06:52.271991 13059 master_runner.cc:263] No leader master found from master de772e1c98c741e39bc06dd11dcbf8b4
I20260620 08:06:52.272042 13059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:06:52.300676 13080 tablet.cc:2369] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:06:52.325222 13305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:06:52.325361 13305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.325641 13305 leader_election.cc:288] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de772e1c98c741e39bc06dd11dcbf8b4 (127.31.115.254:41333), 5f545e67a89147469b0b16b259ed1666 (127.31.115.252:42297)
I20260620 08:06:52.326189 13101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98ec0bc16c24461290d1c2cafe19577a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" is_pre_election: true
I20260620 08:06:52.326345 13101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ec0bc16c24461290d1c2cafe19577a in term 0.
I20260620 08:06:52.326687 13152 leader_election.cc:302] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98ec0bc16c24461290d1c2cafe19577a, de772e1c98c741e39bc06dd11dcbf8b4; no voters:
I20260620 08:06:52.326905 13305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:06:52.326970 13305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:06:52.326992 13305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:52.328008 13305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.328132 13305 leader_election.cc:288] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [CANDIDATE]: Term 1 election: Requested vote from peers de772e1c98c741e39bc06dd11dcbf8b4 (127.31.115.254:41333), 5f545e67a89147469b0b16b259ed1666 (127.31.115.252:42297)
I20260620 08:06:52.328404 13101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98ec0bc16c24461290d1c2cafe19577a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de772e1c98c741e39bc06dd11dcbf8b4"
I20260620 08:06:52.328481 13101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:52.329290 13101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ec0bc16c24461290d1c2cafe19577a in term 1.
I20260620 08:06:52.329514 13152 leader_election.cc:302] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98ec0bc16c24461290d1c2cafe19577a, de772e1c98c741e39bc06dd11dcbf8b4; no voters:
I20260620 08:06:52.329979 13251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98ec0bc16c24461290d1c2cafe19577a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f545e67a89147469b0b16b259ed1666"
I20260620 08:06:52.330116 13251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:52.331024 13251 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ec0bc16c24461290d1c2cafe19577a in term 1.
I20260620 08:06:52.331208 13252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98ec0bc16c24461290d1c2cafe19577a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f545e67a89147469b0b16b259ed1666" is_pre_election: true
I20260620 08:06:52.331278 13252 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 98ec0bc16c24461290d1c2cafe19577a in term 1. Re-sending same reply.
I20260620 08:06:52.332515 13305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:52.332683 13305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [term 1 LEADER]: Becoming Leader. State: Replica: 98ec0bc16c24461290d1c2cafe19577a, State: Running, Role: LEADER
I20260620 08:06:52.332769 13305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } }
I20260620 08:06:52.333150 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ec0bc16c24461290d1c2cafe19577a. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.333222 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:52.333752 13334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:52.333983 13334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:06:52.336231 13252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [term 1 FOLLOWER]: Refusing update from remote peer 98ec0bc16c24461290d1c2cafe19577a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:52.336673 13305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:52.338374 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ec0bc16c24461290d1c2cafe19577a. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.338492 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:06:52.338979 13101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [term 1 FOLLOWER]: Refusing update from remote peer 98ec0bc16c24461290d1c2cafe19577a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:52.339278 13305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [LEADER]: Connected to new peer: Peer: permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:52.340483 13319 sys_catalog.cc:455] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ec0bc16c24461290d1c2cafe19577a. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.340577 13319 sys_catalog.cc:458] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:06:52.342008 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.342096 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:52.342633 13319 sys_catalog.cc:455] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.342717 13319 sys_catalog.cc:458] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:06:52.342994 13335 mvcc.cc:205] Tried to move back new op lower bound from 7298837759326326784 to 7298837759315435520. Current Snapshot: MvccSnapshot[applied={T|T < 7298837759326326784}]
I20260620 08:06:52.343230 13338 mvcc.cc:205] Tried to move back new op lower bound from 7298837759326326784 to 7298837759315435520. Current Snapshot: MvccSnapshot[applied={T|T < 7298837759326326784}]
I20260620 08:06:52.344146 13305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.344226 13305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:52.344787 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98ec0bc16c24461290d1c2cafe19577a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de772e1c98c741e39bc06dd11dcbf8b4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 41333 } } peers { permanent_uuid: "98ec0bc16c24461290d1c2cafe19577a" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 42245 } } peers { permanent_uuid: "5f545e67a89147469b0b16b259ed1666" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 42297 } } }
I20260620 08:06:52.344880 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:06:52.346652 13336 mvcc.cc:205] Tried to move back new op lower bound from 7298837759326326784 to 7298837759315435520. Current Snapshot: MvccSnapshot[applied={T|T < 7298837759326326784}]
I20260620 08:06:52.347888 13334 catalog_manager.cc:1347] Generated new cluster ID: 031a2ee64e1f48278b265793ce79705e
I20260620 08:06:52.347952 13334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:06:52.357272 13334 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:06:52.358124 13334 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:06:52.367709 13334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ec0bc16c24461290d1c2cafe19577a: Generated new TSK 0
I20260620 08:06:52.368186 13334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:06:53.212039 13302 catalog_manager.cc:1259] Loaded cluster ID: 031a2ee64e1f48278b265793ce79705e
I20260620 08:06:53.212123 13302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: loading cluster ID for follower catalog manager: success
I20260620 08:06:53.213014 13302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: acquiring CA information for follower catalog manager: success
I20260620 08:06:53.213359 13302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f545e67a89147469b0b16b259ed1666: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:06:53.269706 13330 catalog_manager.cc:1259] Loaded cluster ID: 031a2ee64e1f48278b265793ce79705e
I20260620 08:06:53.269789 13330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: loading cluster ID for follower catalog manager: success
I20260620 08:06:53.270665 13330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: acquiring CA information for follower catalog manager: success
I20260620 08:06:53.271032 13330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de772e1c98c741e39bc06dd11dcbf8b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:06:53.434307 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13059
I20260620 08:06:53.442680 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13134
I20260620 08:06:53.448117 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13210
2026-06-20T08:06:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2983 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:06:53 dist-test-slave-c4w8 krb5kdc[13353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:06:53 dist-test-slave-c4w8 krb5kdc[13353](info): set up 2 sockets
Jun 20 08:06:53 dist-test-slave-c4w8 krb5kdc[13353](info): commencing operation
krb5kdc: starting...
W20260620 08:06:54.558351 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:06:54 dist-test-slave-c4w8 krb5kdc[13353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:06:54.699149 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:42151
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:54.789955 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:54.790189 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:54.790212 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:54.791395 13366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:54.791436 13366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:54.791451 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:54.791463 13366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:54.791476 13366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:54.793104 13366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:42151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:54.793319 13366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:54.793494 13366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:54.798905 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:54.798959 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:54.798908 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:54.799324 13366 server_base.cc:1034] running on GCE node
I20260620 08:06:54.799552 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:54.799809 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:54.800987 13366 hybrid_clock.cc:648] HybridClock initialized: now 1781942814800972 us; error 35 us; skew 500 ppm
Jun 20 08:06:54 dist-test-slave-c4w8 krb5kdc[13353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942814, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:54.803686 13366 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:54.804105 13366 webserver.cc:460] Webserver started at http://127.31.115.254:34709/ using document root <none> and password file <none>
I20260620 08:06:54.804353 13366 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:54.804401 13366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:54.804548 13366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:54.805722 13366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "a211f00912a54963b270b31e566f7e7b"
format_stamp: "Formatted at 2026-06-20 08:06:54 on dist-test-slave-c4w8"
server_key: "7d29e3e2c86f91449f553509ba05c089"
server_key_iv: "bc81982ecfc859746bfd6b65a10191f9"
server_key_version: "encryptionkey@0"
I20260620 08:06:54.806102 13366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "a211f00912a54963b270b31e566f7e7b"
format_stamp: "Formatted at 2026-06-20 08:06:54 on dist-test-slave-c4w8"
server_key: "7d29e3e2c86f91449f553509ba05c089"
server_key_iv: "bc81982ecfc859746bfd6b65a10191f9"
server_key_version: "encryptionkey@0"
I20260620 08:06:54.807615 13366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:06:54.808550 13381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:54.808836 13366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:54.808938 13366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "a211f00912a54963b270b31e566f7e7b"
format_stamp: "Formatted at 2026-06-20 08:06:54 on dist-test-slave-c4w8"
server_key: "7d29e3e2c86f91449f553509ba05c089"
server_key_iv: "bc81982ecfc859746bfd6b65a10191f9"
server_key_version: "encryptionkey@0"
I20260620 08:06:54.809041 13366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:54.859397 13366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:54.861961 13366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:54.862128 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:54.866173 13366 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:42151
I20260620 08:06:54.866248 13433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:42151 every 8 connection(s)
I20260620 08:06:54.866590 13366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:54.867655 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:54.870177 13434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b: Bootstrap starting.
I20260620 08:06:54.870801 13434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:54.871122 13434 log.cc:824] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:54.871968 13434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b: No bootstrap required, opened a new log
I20260620 08:06:54.873682 13434 raft_consensus.cc:348] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211f00912a54963b270b31e566f7e7b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42151 } }
I20260620 08:06:54.873867 13434 raft_consensus.cc:374] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:54.873919 13434 raft_consensus.cc:729] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a211f00912a54963b270b31e566f7e7b, State: Initialized, Role: FOLLOWER
I20260620 08:06:54.874099 13434 consensus_queue.cc:260] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211f00912a54963b270b31e566f7e7b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42151 } }
I20260620 08:06:54.874184 13434 raft_consensus.cc:388] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:06:54.874231 13434 raft_consensus.cc:482] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:06:54.874281 13434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:54.874847 13434 raft_consensus.cc:504] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211f00912a54963b270b31e566f7e7b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42151 } }
I20260620 08:06:54.874974 13434 leader_election.cc:302] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a211f00912a54963b270b31e566f7e7b; no voters:
I20260620 08:06:54.875174 13434 leader_election.cc:288] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:06:54.875310 13437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:54.875487 13434 sys_catalog.cc:564] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:54.875736 13437 raft_consensus.cc:686] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [term 1 LEADER]: Becoming Leader. State: Replica: a211f00912a54963b270b31e566f7e7b, State: Running, Role: LEADER
I20260620 08:06:54.876065 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13366
I20260620 08:06:54.876194 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:06:54.876245 13437 consensus_queue.cc:237] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211f00912a54963b270b31e566f7e7b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42151 } }
I20260620 08:06:54.876431 32207 external_mini_cluster.cc:1428] Setting key 5703c9c8e245bb6eb57f1f23902feaa3
I20260620 08:06:54.877019 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a211f00912a54963b270b31e566f7e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211f00912a54963b270b31e566f7e7b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42151 } } }
I20260620 08:06:54.877167 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:54.877465 13439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a211f00912a54963b270b31e566f7e7b. Latest consensus state: current_term: 1 leader_uuid: "a211f00912a54963b270b31e566f7e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a211f00912a54963b270b31e566f7e7b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42151 } } }
I20260620 08:06:54.877691 13439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b [sys.catalog]: This master's current role is: LEADER
W20260620 08:06:54.877753 13452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:06:54.877801 13452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:06:54.877861 13453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:54.878008 13453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:06:54.881032 13453 catalog_manager.cc:1347] Generated new cluster ID: ec114d60bffc44bdafa37dd4d70b1646
I20260620 08:06:54.881176 13453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:06:54 dist-test-slave-c4w8 krb5kdc[13353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:54.889256 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:54.877195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.31.115.254:42151)
0620 08:06:54.877629 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:54.877639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:54.877674 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:06:54.877810 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:54.877818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:54.877882 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:54.878054 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:54.878804 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:54.880284 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:54.881368 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:54.882086 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:54.882177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:54.885188 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:54.885194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:54.885194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:54.885209 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:54.887413 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:54.887876 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:54.887881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:54.887881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:54.888012 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:54.888247 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:54.888248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:54.888248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:54.888556 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:54.888732 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:54.888893 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 08:06:54.889055 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":81,"threads_started":1}
I20260620 08:06:54.890528 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:06:54.892053 13453 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:06:54.892846 13453 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:06:54.901042 13453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a211f00912a54963b270b31e566f7e7b: Generated new TSK 0
I20260620 08:06:54.901300 13453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:06:54.944854 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13366
2026-06-20T08:06:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1494 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:06:55 dist-test-slave-c4w8 krb5kdc[13461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:06:55 dist-test-slave-c4w8 krb5kdc[13461](info): set up 2 sockets
Jun 20 08:06:55 dist-test-slave-c4w8 krb5kdc[13461](info): commencing operation
krb5kdc: starting...
W20260620 08:06:56.054189 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:06:56.181542 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40507
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:56.266240 13474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:56.266448 13474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:56.266486 13474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:56.267717 13474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:56.267762 13474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:56.267777 13474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:56.267791 13474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:56.267802 13474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:56.269392 13474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:56.269682 13474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:56.269948 13474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:56.275005 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.275106 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.275157 13479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:56.275449 13474 server_base.cc:1034] running on GCE node
I20260620 08:06:56.275743 13474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:56.276067 13474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:56.277247 13474 hybrid_clock.cc:648] HybridClock initialized: now 1781942816277243 us; error 59 us; skew 500 ppm
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:56.280197 13474 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:56.280714 13474 webserver.cc:460] Webserver started at http://127.31.115.254:37155/ using document root <none> and password file <none>
I20260620 08:06:56.281000 13474 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:56.281071 13474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:56.281222 13474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:56.282366 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "43b1c329edfd4379ba3052ab534400e2"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "adc714d666c63f3d24b36b9a0fb0bb13"
server_key_iv: "fca6c28ce58a0b25727c9503954719f5"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.282727 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "43b1c329edfd4379ba3052ab534400e2"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "adc714d666c63f3d24b36b9a0fb0bb13"
server_key_iv: "fca6c28ce58a0b25727c9503954719f5"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.284173 13474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:06:56.285014 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.285244 13474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:06:56.285406 13474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "43b1c329edfd4379ba3052ab534400e2"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "adc714d666c63f3d24b36b9a0fb0bb13"
server_key_iv: "fca6c28ce58a0b25727c9503954719f5"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.285488 13474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:56.295985 13474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:56.298298 13474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:56.298470 13474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:56.302939 13474 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:40507
I20260620 08:06:56.303128 13541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:40507 every 8 connection(s)
I20260620 08:06:56.303414 13474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:56.304448 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:56.306793 13542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap starting.
I20260620 08:06:56.306896 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13474
I20260620 08:06:56.307060 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:06:56.307268 32207 external_mini_cluster.cc:1428] Setting key 87ed3efc4cec15170e9941b0259a9139
I20260620 08:06:56.307574 13542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:56.307936 13542 log.cc:824] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:56.308784 13542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: No bootstrap required, opened a new log
I20260620 08:06:56.311149 13542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:56.311331 13542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:56.311360 13542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b1c329edfd4379ba3052ab534400e2, State: Initialized, Role: FOLLOWER
I20260620 08:06:56.311491 13542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:56.311587 13542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:06:56.311611 13542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:06:56.311679 13542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:56.312299 13542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:56.312407 13542 leader_election.cc:302] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b1c329edfd4379ba3052ab534400e2; no voters:
I20260620 08:06:56.312554 13542 leader_election.cc:288] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:06:56.312614 13547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:56.312819 13547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 LEADER]: Becoming Leader. State: Replica: 43b1c329edfd4379ba3052ab534400e2, State: Running, Role: LEADER
I20260620 08:06:56.312986 13547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:56.312825 13542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:56.313333 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b1c329edfd4379ba3052ab534400e2. Latest consensus state: current_term: 1 leader_uuid: "43b1c329edfd4379ba3052ab534400e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } } }
I20260620 08:06:56.313411 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:56.313372 13548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43b1c329edfd4379ba3052ab534400e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } } }
I20260620 08:06:56.313597 13548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:56.313766 13551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:56.314301 13551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:56.316254 13551 catalog_manager.cc:1347] Generated new cluster ID: 481f982a25fb4a52828ad1506906f614
I20260620 08:06:56.316304 13551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:06:56.319087 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.307845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.31.115.254:40507)
0620 08:06:56.308069 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.308079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.308113 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:06:56.308737 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.308746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.308809 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.309001 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:56.309634 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.311118 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.312170 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.312811 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.312928 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.315833 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:56.315837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:56.315837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:56.315849 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:56.317478 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.317907 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.317910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.317911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.318001 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.318230 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.318231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.318231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.318470 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:56.318664 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.318731 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.318868 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260620 08:06:56.345012 13551 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:06:56.345602 13551 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:06:56.364586 13551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Generated new TSK 0
I20260620 08:06:56.364828 13551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:06:56.396417 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:40507
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:56.480861 13568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:56.481178 13568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:56.481223 13568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:56.482517 13568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:56.482600 13568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:56.482674 13568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:06:56.484246 13568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:56.484534 13568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:56.484777 13568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:56.489957 13576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.489957 13573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.490000 13574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:56.490091 13568 server_base.cc:1034] running on GCE node
I20260620 08:06:56.490434 13568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:56.490728 13568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:56.491923 13568 hybrid_clock.cc:648] HybridClock initialized: now 1781942816491909 us; error 30 us; skew 500 ppm
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:56.494637 13568 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:06:56.495164 13568 webserver.cc:460] Webserver started at http://127.31.115.193:46003/ using document root <none> and password file <none>
I20260620 08:06:56.495429 13568 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:56.495507 13568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:56.495671 13568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:56.496886 13568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "b0250a0fd7c04c14888dfc80a5afc641"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "d581f8ea5d826a80b1f84972d8288422"
server_key_iv: "e369e2c28f60ac52c727a036544085b3"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.497294 13568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "b0250a0fd7c04c14888dfc80a5afc641"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "d581f8ea5d826a80b1f84972d8288422"
server_key_iv: "e369e2c28f60ac52c727a036544085b3"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.498867 13568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:06:56.499758 13583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.500015 13568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:56.500118 13568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "b0250a0fd7c04c14888dfc80a5afc641"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "d581f8ea5d826a80b1f84972d8288422"
server_key_iv: "e369e2c28f60ac52c727a036544085b3"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.500207 13568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:56.517292 13568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:56.519753 13568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:56.519953 13568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:56.520221 13568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:56.520555 13568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:56.520610 13568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.520651 13568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:56.520699 13568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.526964 13568 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:35933
I20260620 08:06:56.527011 13696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:35933 every 8 connection(s)
I20260620 08:06:56.527352 13568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:06:56.531963 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13568
I20260620 08:06:56.532065 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:06:56.532285 32207 external_mini_cluster.cc:1428] Setting key ffabd2c077a840aa9bd26358f202ae08
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:56.537971 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.528148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:45115 (local address 127.31.115.254:40507)
0620 08:06:56.528248 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.528250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.528460 (+ 210us) server_negotiation.cc:407] Connection header received
0620 08:06:56.528995 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.528996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.529014 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.529080 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:56.530048 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.530561 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.531735 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.531978 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.532093 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.534529 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:56.534531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:56.534531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:56.534541 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:56.536185 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.536810 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.536812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.536812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.536869 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.537196 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.537197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.537197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.537341 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:56.537398 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.537819 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.537898 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 08:06:56.538353 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.528117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:45115)
0620 08:06:56.528301 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.528348 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.528888 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.529134 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.529138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.529739 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:56.529943 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.529948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.530672 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.530672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.531635 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.531639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.532164 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.532165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.532353 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.532984 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:56.532996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:56.534451 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:56.536276 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:56.536285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:56.536288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:56.536713 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:56.536944 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:56.536945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:56.536945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:56.537113 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:56.537422 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:56.537430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:56.537659 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.537802 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.538011 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260620 08:06:56.539089 13697 heartbeater.cc:347] Connected to a master server at 127.31.115.254:40507
I20260620 08:06:56.539192 13697 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:56.539402 13697 heartbeater.cc:511] Master 127.31.115.254:40507 requested a full tablet report, sending...
I20260620 08:06:56.539988 13506 ts_manager.cc:194] Registered new tserver with Master: b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933)
I20260620 08:06:56.540602 13506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:45115
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:06:56.561470 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:40507
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:56.643488 13702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:56.643700 13702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:56.643725 13702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:56.644819 13702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:56.644851 13702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:56.644877 13702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:06:56.646353 13702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:56.646605 13702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:56.646787 13702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:56.652132 13710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:56.652236 13702 server_base.cc:1034] running on GCE node
W20260620 08:06:56.652146 13708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.652177 13707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:56.652588 13702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:56.652851 13702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:56.654021 13702 hybrid_clock.cc:648] HybridClock initialized: now 1781942816653990 us; error 43 us; skew 500 ppm
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:56.656805 13702 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:06:56.657334 13702 webserver.cc:460] Webserver started at http://127.31.115.194:40443/ using document root <none> and password file <none>
I20260620 08:06:56.657577 13702 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:56.657672 13702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:56.657827 13702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:56.658970 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "6e3b9581f25544ea971223a561571e07"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "f68f61d350c39e6f580dfc8a464f4fe9"
server_key_iv: "3fb855bc162bbbed142f388c120c24fe"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.659340 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3b9581f25544ea971223a561571e07"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "f68f61d350c39e6f580dfc8a464f4fe9"
server_key_iv: "3fb855bc162bbbed142f388c120c24fe"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.660723 13702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:06:56.661553 13717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.661787 13702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:06:56.661902 13702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "6e3b9581f25544ea971223a561571e07"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "f68f61d350c39e6f580dfc8a464f4fe9"
server_key_iv: "3fb855bc162bbbed142f388c120c24fe"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.661991 13702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:56.673516 13702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:56.676201 13702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:56.676556 13702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:56.676848 13702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:56.677208 13702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:56.677266 13702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.677309 13702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:56.677350 13702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.683491 13702 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:46307
I20260620 08:06:56.683672 13830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:46307 every 8 connection(s)
I20260620 08:06:56.683920 13702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:06:56.686817 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13702
I20260620 08:06:56.686950 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:06:56.687131 32207 external_mini_cluster.cc:1428] Setting key dca54bf97ae9b4457227d6a06c6565c3
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:56.695026 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.684714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:33307 (local address 127.31.115.254:40507)
0620 08:06:56.684820 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.684823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.685002 (+ 179us) server_negotiation.cc:407] Connection header received
0620 08:06:56.685628 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.685630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.685643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.685692 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:56.686732 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.687406 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.688972 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.689176 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.689241 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.691658 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:56.691661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:56.691661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:56.691671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:56.693432 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.693932 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.693934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.693935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.694004 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.694278 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.694279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.694279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.694443 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:56.694529 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.694795 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.694893 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 08:06:56.695065 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.684616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:33307)
0620 08:06:56.684853 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.684903 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.685463 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.685762 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.685766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.686408 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:56.686630 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.686635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.687494 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.687495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.688819 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.688822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.689272 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.689273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.689440 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.690117 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:56.690127 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:56.691550 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:56.693536 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:56.693544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:56.693547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:56.693849 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:56.694087 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:56.694088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:56.694088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:56.694206 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:56.694516 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:56.694522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:56.694651 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.694740 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.694870 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
I20260620 08:06:56.695899 13831 heartbeater.cc:347] Connected to a master server at 127.31.115.254:40507
I20260620 08:06:56.696020 13831 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:56.696251 13831 heartbeater.cc:511] Master 127.31.115.254:40507 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:06:56.696754 13506 ts_manager.cc:194] Registered new tserver with Master: 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307)
I20260620 08:06:56.697155 13506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:33307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:06:56.717402 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:40507
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:56.800765 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:56.801028 13836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:56.801070 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:56.802315 13836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:56.802384 13836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:56.802434 13836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:06:56.804129 13836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:56.804417 13836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:56.804661 13836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:56.809931 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.809937 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:56.809950 13841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:56.810124 13836 server_base.cc:1034] running on GCE node
I20260620 08:06:56.810411 13836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:56.810698 13836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:56.811866 13836 hybrid_clock.cc:648] HybridClock initialized: now 1781942816811873 us; error 46 us; skew 500 ppm
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:56.819957 13836 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:06:56.820470 13836 webserver.cc:460] Webserver started at http://127.31.115.195:34473/ using document root <none> and password file <none>
I20260620 08:06:56.820683 13836 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:56.820725 13836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:56.820818 13836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:06:56.821942 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "b83a8abe1d7f401da80d6f2674abffa1"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "e90a1e9a95aaf35f5342bea6221cce61"
server_key_iv: "0cce43de13d42382af587642cb9838d1"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.822247 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "b83a8abe1d7f401da80d6f2674abffa1"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "e90a1e9a95aaf35f5342bea6221cce61"
server_key_iv: "0cce43de13d42382af587642cb9838d1"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.823760 13836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:06:56.824688 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.824937 13836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:56.825070 13836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "b83a8abe1d7f401da80d6f2674abffa1"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "e90a1e9a95aaf35f5342bea6221cce61"
server_key_iv: "0cce43de13d42382af587642cb9838d1"
server_key_version: "encryptionkey@0"
I20260620 08:06:56.825155 13836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:56.839241 13836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:56.841784 13836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:56.841960 13836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:56.842219 13836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:56.842561 13836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:56.842618 13836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.842665 13836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:56.842700 13836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:56.848783 13836 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:44691
I20260620 08:06:56.848847 13964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:44691 every 8 connection(s)
I20260620 08:06:56.849159 13836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:06:56.852877 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 13836
I20260620 08:06:56.852983 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:06:56.853199 32207 external_mini_cluster.cc:1428] Setting key c32034b0bf80d9757968948c0836e44b
Jun 20 08:06:56 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:56.859061 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.849931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:34597 (local address 127.31.115.254:40507)
0620 08:06:56.850020 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.850022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.850215 (+ 193us) server_negotiation.cc:407] Connection header received
0620 08:06:56.850758 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.850759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.850771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.850829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:56.851831 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.852341 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.853763 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.854055 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.854153 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.856422 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:56.856423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:56.856424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:56.856431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:56.857842 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.858252 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.858253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.858253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.858303 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.858516 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.858517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.858517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.858622 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:56.858672 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.858929 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.858992 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:06:56.859341 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.849918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:34597)
0620 08:06:56.850084 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.850130 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.850651 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.850902 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.850906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.851508 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:56.851713 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.851718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.852446 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.852448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.853554 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.853557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.854175 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.854176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.854323 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.854907 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:56.854917 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:56.856325 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:56.857935 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:56.857942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:56.857944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:56.858187 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:56.858362 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:56.858362 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:56.858362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:56.858463 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:56.858678 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:56.858683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:56.858805 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.858892 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.859035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260620 08:06:56.859970 13965 heartbeater.cc:347] Connected to a master server at 127.31.115.254:40507
I20260620 08:06:56.860067 13965 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:56.860229 13965 heartbeater.cc:511] Master 127.31.115.254:40507 requested a full tablet report, sending...
I20260620 08:06:56.860636 13506 ts_manager.cc:194] Registered new tserver with Master: b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
I20260620 08:06:56.860983 13506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:34597
I20260620 08:06:56.865355 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:06:56.874152 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.866915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.31.115.254:40507)
0620 08:06:56.866996 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.866998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.867115 (+ 117us) server_negotiation.cc:407] Connection header received
0620 08:06:56.867156 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.867157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.867169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.867226 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:56.867809 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.868423 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.869187 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.869403 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.869461 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.871055 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:56.871056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:56.871057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:56.871063 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:56.872614 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.873062 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.873064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.873064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.873124 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:56.873412 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:56.873413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:56.873414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:56.873583 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:56.873756 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.873998 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.874077 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:06:56.876144 13506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:06:56.877890 13506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:06:56.885030 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.881018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.31.115.194:46307)
0620 08:06:56.881428 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.881432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.881441 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:06:56.881477 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.881479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.881494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.881566 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:56.881682 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.882808 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884021 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.884568 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884658 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.884720 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.884884 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.884927 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":88,"threads_started":1}
I20260620 08:06:56.885031 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.880774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:46307 (local address 127.0.0.1:37400)
0620 08:06:56.881003 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.881016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.881074 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.881576 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.881576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.881578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:56.881632 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.881633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.882950 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.882951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.883922 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.883925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.884669 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.884750 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.884764 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.884865 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.884927 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":121,"threads_started":1}
I20260620 08:06:56.885210 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.880702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.31.115.193:35933)
0620 08:06:56.881204 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.881209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.881219 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:06:56.881266 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.881267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.881281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.881374 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:56.881582 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.882363 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.883825 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.884592 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884667 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.884719 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.885079 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.885126 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":75,"threads_started":1}
I20260620 08:06:56.885293 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.880942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.31.115.195:44691)
0620 08:06:56.881257 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.881262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.881272 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:06:56.881313 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.881315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.881331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.881657 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:56.881734 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.882812 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884139 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.884808 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.885082 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.885142 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.885166 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.885203 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":87,"threads_started":1}
I20260620 08:06:56.885747 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.880693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44691 (local address 127.0.0.1:33388)
0620 08:06:56.880954 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.880965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.881135 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.881432 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.881433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.881435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:56.881543 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.881546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.882925 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.882926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.884056 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.884058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.884919 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.884972 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.884975 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.885005 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.885039 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":68,"thread_start_us":88,"threads_started":1}
I20260620 08:06:56.885039 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.880536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35933 (local address 127.0.0.1:44258)
0620 08:06:56.880675 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.880721 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.880786 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.881409 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.881412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.881414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:56.881493 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.881496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.882515 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.882516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.883716 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.883719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.884756 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.884757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.884825 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.884830 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.884879 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.884930 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":89,"threads_started":1}
I20260620 08:06:56.886176 13765 tablet_service.cc:1467] Processing CreateTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 (DEFAULT_TABLE table=test-table [id=e92b909463204ce7a9f0375bde0babd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:56.886567 13765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8045de2fa5584ccea690bd96c98d9e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:56.886550 13899 tablet_service.cc:1467] Processing CreateTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 (DEFAULT_TABLE table=test-table [id=e92b909463204ce7a9f0375bde0babd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:56.886911 13899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8045de2fa5584ccea690bd96c98d9e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:56.889946 13982 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Bootstrap starting.
I20260620 08:06:56.890455 13631 tablet_service.cc:1467] Processing CreateTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 (DEFAULT_TABLE table=test-table [id=e92b909463204ce7a9f0375bde0babd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:56.890643 13982 tablet_bootstrap.cc:654] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:56.890806 13631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8045de2fa5584ccea690bd96c98d9e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:56.891253 13982 log.cc:824] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:56.891741 13983 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Bootstrap starting.
I20260620 08:06:56.892300 13983 tablet_bootstrap.cc:654] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:56.892830 13983 log.cc:824] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:56.893558 13983 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: No bootstrap required, opened a new log
I20260620 08:06:56.893668 13983 ts_tablet_manager.cc:1397] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:06:56.894320 13982 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: No bootstrap required, opened a new log
I20260620 08:06:56.894385 13982 ts_tablet_manager.cc:1397] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:06:56.895215 13983 raft_consensus.cc:348] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.895375 13983 raft_consensus.cc:374] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:56.895423 13983 raft_consensus.cc:729] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83a8abe1d7f401da80d6f2674abffa1, State: Initialized, Role: FOLLOWER
I20260620 08:06:56.895568 13983 consensus_queue.cc:260] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.895864 13983 ts_tablet_manager.cc:1428] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:06:56.895901 13965 heartbeater.cc:503] Master 127.31.115.254:40507 was elected leader, sending a full tablet report...
I20260620 08:06:56.896096 13986 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap starting.
I20260620 08:06:56.896006 13982 raft_consensus.cc:348] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.896158 13982 raft_consensus.cc:374] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:56.896205 13982 raft_consensus.cc:729] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3b9581f25544ea971223a561571e07, State: Initialized, Role: FOLLOWER
I20260620 08:06:56.896327 13982 consensus_queue.cc:260] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.896620 13982 ts_tablet_manager.cc:1428] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:06:56.896643 13831 heartbeater.cc:503] Master 127.31.115.254:40507 was elected leader, sending a full tablet report...
I20260620 08:06:56.896811 13986 tablet_bootstrap.cc:654] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:56.897418 13986 log.cc:824] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:56.898237 13986 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: No bootstrap required, opened a new log
I20260620 08:06:56.898309 13986 ts_tablet_manager.cc:1397] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:06:56.899677 13986 raft_consensus.cc:348] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.899811 13986 raft_consensus.cc:374] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:56.899833 13986 raft_consensus.cc:729] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0250a0fd7c04c14888dfc80a5afc641, State: Initialized, Role: FOLLOWER
I20260620 08:06:56.899966 13986 consensus_queue.cc:260] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.900250 13986 ts_tablet_manager.cc:1428] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:06:56.900315 13697 heartbeater.cc:503] Master 127.31.115.254:40507 was elected leader, sending a full tablet report...
I20260620 08:06:56.920746 13988 raft_consensus.cc:482] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:06:56.920897 13988 raft_consensus.cc:504] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.921242 13988 leader_election.cc:288] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933), b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
I20260620 08:06:56.924360 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.921509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:39175 (local address 127.31.115.193:35933)
0620 08:06:56.921559 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.921562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.921837 (+ 275us) server_negotiation.cc:407] Connection header received
0620 08:06:56.921878 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.921879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.921888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.921955 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:56.922126 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.922601 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.923597 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.923999 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.924082 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.924093 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.924249 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.924287 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:06:56.924427 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.921534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:59393 (local address 127.31.115.195:44691)
0620 08:06:56.921584 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.921585 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.921664 (+ 79us) server_negotiation.cc:407] Connection header received
0620 08:06:56.921696 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.921697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.921710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.921768 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:56.922091 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.922662 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.923619 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.924028 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.924126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.924136 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.924313 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.924355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:06:56.924427 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.921446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44691 (local address 127.31.115.194:59393)
0620 08:06:56.921510 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.921522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.921577 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.921861 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.921862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.921864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:56.921940 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.921942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.922761 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.922763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.923509 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.923512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.924125 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.924126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.924212 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.924216 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.924291 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.924336 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:06:56.924620 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.921447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35933 (local address 127.31.115.194:39175)
0620 08:06:56.921691 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.921703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:56.921769 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:56.921961 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:56.921962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:56.921964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:56.922050 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.922052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.922693 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.922694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.923496 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:56.923498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.924092 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.924093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:56.924170 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.924174 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:06:56.924239 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:06:56.924280 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":15,"thread_start_us":167,"threads_started":1}
I20260620 08:06:56.924835 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8045de2fa5584ccea690bd96c98d9e75" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83a8abe1d7f401da80d6f2674abffa1" is_pre_election: true
I20260620 08:06:56.924926 13651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8045de2fa5584ccea690bd96c98d9e75" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0250a0fd7c04c14888dfc80a5afc641" is_pre_election: true
I20260620 08:06:56.925063 13651 raft_consensus.cc:2454] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 0.
I20260620 08:06:56.925062 13919 raft_consensus.cc:2454] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 0.
I20260620 08:06:56.925338 13721 leader_election.cc:302] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b9581f25544ea971223a561571e07, b0250a0fd7c04c14888dfc80a5afc641; no voters:
I20260620 08:06:56.925521 13988 raft_consensus.cc:2781] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:06:56.925585 13988 raft_consensus.cc:482] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:06:56.925633 13988 raft_consensus.cc:3037] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:56.926329 13988 raft_consensus.cc:504] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.926492 13988 leader_election.cc:288] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 election: Requested vote from peers b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933), b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
I20260620 08:06:56.926663 13651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8045de2fa5584ccea690bd96c98d9e75" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0250a0fd7c04c14888dfc80a5afc641"
I20260620 08:06:56.926667 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8045de2fa5584ccea690bd96c98d9e75" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83a8abe1d7f401da80d6f2674abffa1"
I20260620 08:06:56.926756 13919 raft_consensus.cc:3037] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:56.926757 13651 raft_consensus.cc:3037] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:56.927706 13651 raft_consensus.cc:2454] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 1.
I20260620 08:06:56.927706 13919 raft_consensus.cc:2454] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 1.
I20260620 08:06:56.927927 13721 leader_election.cc:302] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b9581f25544ea971223a561571e07, b0250a0fd7c04c14888dfc80a5afc641; no voters:
I20260620 08:06:56.928033 13988 raft_consensus.cc:2781] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:56.928243 13988 raft_consensus.cc:686] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 1 LEADER]: Becoming Leader. State: Replica: 6e3b9581f25544ea971223a561571e07, State: Running, Role: LEADER
I20260620 08:06:56.928399 13988 consensus_queue.cc:237] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:56.929366 13505 catalog_manager.cc:5495] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3b9581f25544ea971223a561571e07 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "6e3b9581f25544ea971223a561571e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } health_report { overall_health: UNKNOWN } } }
W20260620 08:06:56.934558 13832 tablet.cc:2369] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:06:56.946312 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.943647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.31.115.194:46307)
0620 08:06:56.943727 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:56.943729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:56.943739 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:06:56.943776 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:56.943777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:56.943797 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:56.943909 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:06:56.944073 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.944603 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.945399 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:56.945592 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:56.945749 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:56.945946 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:06:56.946061 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:06:56.946162 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:56.946179 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 08:06:56.946201 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:06:56.948413 13919 raft_consensus.cc:1261] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Refusing update from remote peer 6e3b9581f25544ea971223a561571e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:56.948414 13651 raft_consensus.cc:1261] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Refusing update from remote peer 6e3b9581f25544ea971223a561571e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:56.948810 13988 consensus_queue.cc:1035] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:56.949021 13992 consensus_queue.cc:1035] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:56.952278 13996 mvcc.cc:205] Tried to move back new op lower bound from 7298837778217349120 to 7298837778139348992. Current Snapshot: MvccSnapshot[applied={T|T < 7298837778217349120}]
I20260620 08:06:56.954790 13506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:06:56.954910 13506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:06:56.955768 13506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Sending DeleteTablet for 3 replicas of tablet 8045de2fa5584ccea690bd96c98d9e75
I20260620 08:06:56.956130 13899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:56 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:33388
I20260620 08:06:56.956139 13631 tablet_service.cc:1514] Processing DeleteTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:56 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:44258
I20260620 08:06:56.956166 13765 tablet_service.cc:1514] Processing DeleteTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:56 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:37400
I20260620 08:06:56.956423 14006 tablet_replica.cc:330] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: stopping tablet replica
I20260620 08:06:56.956442 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13568
I20260620 08:06:56.956565 14007 tablet_replica.cc:330] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: stopping tablet replica
I20260620 08:06:56.957717 14005 tablet_replica.cc:330] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: stopping tablet replica
I20260620 08:06:56.957825 14005 raft_consensus.cc:2229] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:06:56.957968 14005 raft_consensus.cc:2258] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:56.958442 14005 ts_tablet_manager.cc:1905] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:06:56.960196 14005 ts_tablet_manager.cc:1918] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:56.960273 14005 log.cc:1196] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/8045de2fa5584ccea690bd96c98d9e75
I20260620 08:06:56.960546 14005 ts_tablet_manager.cc:1939] T 8045de2fa5584ccea690bd96c98d9e75 P 6e3b9581f25544ea971223a561571e07: Deleting consensus metadata
I20260620 08:06:56.961046 13493 catalog_manager.cc:4841] TS 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307): tablet 8045de2fa5584ccea690bd96c98d9e75 (table test-table [id=e92b909463204ce7a9f0375bde0babd7]) successfully deleted
I20260620 08:06:56.962242 14006 raft_consensus.cc:2229] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:06:56.962391 14006 raft_consensus.cc:2258] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:56.963184 14006 ts_tablet_manager.cc:1905] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:06:56.963517 13493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:35933 (error 108)
I20260620 08:06:56.963877 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13702
I20260620 08:06:56.964888 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:56.963956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35933 (local address 127.0.0.1:44270)
0620 08:06:56.964810 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:56.964856 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:35933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":828}
W20260620 08:06:56.965056 13493 catalog_manager.cc:4584] TS b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8045de2fa5584ccea690bd96c98d9e75: Network error: Client connection negotiation failed: client connection to 127.31.115.193:35933: connect: Connection refused (error 111)
I20260620 08:06:56.966840 14006 ts_tablet_manager.cc:1918] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:56.966938 14006 log.cc:1196] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/8045de2fa5584ccea690bd96c98d9e75
I20260620 08:06:56.967209 14006 ts_tablet_manager.cc:1939] T 8045de2fa5584ccea690bd96c98d9e75 P b83a8abe1d7f401da80d6f2674abffa1: Deleting consensus metadata
I20260620 08:06:56.967729 13491 catalog_manager.cc:4841] TS b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691): tablet 8045de2fa5584ccea690bd96c98d9e75 (table test-table [id=e92b909463204ce7a9f0375bde0babd7]) successfully deleted
I20260620 08:06:56.970736 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13836
I20260620 08:06:56.976173 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 13474
I20260620 08:06:56.982180 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40507
--webserver_interface=127.31.115.254
--webserver_port=37155
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:57.068715 14009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:57.068923 14009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:57.068948 14009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:57.070250 14009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:57.070297 14009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:57.070310 14009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:57.070323 14009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:57.070336 14009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:57.071919 14009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=37155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:57.072216 14009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:57.072641 14009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:57.083892 14016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.083842 14018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.083998 14009 server_base.cc:1034] running on GCE node
W20260620 08:06:57.083853 14015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.084479 14009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:57.084861 14009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:57.086026 14009 hybrid_clock.cc:648] HybridClock initialized: now 1781942817085998 us; error 39 us; skew 500 ppm
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:57.088894 14009 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:57.089435 14009 webserver.cc:460] Webserver started at http://127.31.115.254:37155/ using document root <none> and password file <none>
I20260620 08:06:57.089780 14009 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:57.089869 14009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:57.091378 14009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:57.092188 14025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.092453 14009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:57.092592 14009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "43b1c329edfd4379ba3052ab534400e2"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "adc714d666c63f3d24b36b9a0fb0bb13"
server_key_iv: "fca6c28ce58a0b25727c9503954719f5"
server_key_version: "encryptionkey@0"
I20260620 08:06:57.093016 14009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:57.104897 14009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:57.107486 14009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:57.107681 14009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:57.112630 14009 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:40507
I20260620 08:06:57.112665 14077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:40507 every 8 connection(s)
I20260620 08:06:57.113094 14009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:57.115000 14078 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:06:57.115754 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap starting.
I20260620 08:06:57.117841 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14009
I20260620 08:06:57.118186 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:35933
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:40507
--webserver_port=46003
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:57.125988 14078 log.cc:824] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:57.128054 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:06:57.128413 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap complete.
I20260620 08:06:57.131214 14078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.131474 14078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b1c329edfd4379ba3052ab534400e2, State: Initialized, Role: FOLLOWER
I20260620 08:06:57.131657 14078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.131736 14078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:06:57.131790 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:06:57.131840 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:06:57.133010 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.133129 14078 leader_election.cc:302] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b1c329edfd4379ba3052ab534400e2; no voters:
I20260620 08:06:57.133344 14078 leader_election.cc:288] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:06:57.133387 14082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:06:57.133503 14082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 LEADER]: Becoming Leader. State: Replica: 43b1c329edfd4379ba3052ab534400e2, State: Running, Role: LEADER
I20260620 08:06:57.133625 14082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.133705 14078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:57.133927 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43b1c329edfd4379ba3052ab534400e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } } }
I20260620 08:06:57.134024 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:57.133924 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b1c329edfd4379ba3052ab534400e2. Latest consensus state: current_term: 2 leader_uuid: "43b1c329edfd4379ba3052ab534400e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } } }
I20260620 08:06:57.134244 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:57.134557 14086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:57.135486 14086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:06:57.135605 14086 catalog_manager.cc:1259] Loaded cluster ID: 481f982a25fb4a52828ad1506906f614
I20260620 08:06:57.135856 14086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:06:57.136111 14086 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 08:06:57.138860 14086 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 08:06:57.142796 14086 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:06:57.143071 14086 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Loaded TSK: 0
I20260620 08:06:57.143303 14086 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:06:57.229667 14080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:57.229965 14080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:57.229997 14080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:57.231375 14080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:57.231437 14080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:57.231472 14080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:06:57.233192 14080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:35933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=46003
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:57.233510 14080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:57.233872 14080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:57.240507 14106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.240550 14108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.240522 14105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.240882 14080 server_base.cc:1034] running on GCE node
I20260620 08:06:57.241096 14080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:57.241436 14080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:57.242645 14080 hybrid_clock.cc:648] HybridClock initialized: now 1781942817242619 us; error 41 us; skew 500 ppm
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:57.245582 14080 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:06:57.246342 14080 webserver.cc:460] Webserver started at http://127.31.115.193:46003/ using document root <none> and password file <none>
I20260620 08:06:57.246619 14080 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:57.246709 14080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:57.248157 14080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:06:57.248912 14115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.249173 14080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:06:57.249286 14080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "b0250a0fd7c04c14888dfc80a5afc641"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "d581f8ea5d826a80b1f84972d8288422"
server_key_iv: "e369e2c28f60ac52c727a036544085b3"
server_key_version: "encryptionkey@0"
I20260620 08:06:57.249788 14080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:57.274435 14080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:57.277364 14080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:57.277550 14080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:57.277882 14080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:57.278514 14122 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:06:57.280185 14080 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:06:57.280252 14080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:06:57.280319 14080 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:06:57.281224 14080 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:06:57.281270 14080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:06:57.281386 14122 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap starting.
I20260620 08:06:57.289201 14080 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:35933
I20260620 08:06:57.289762 14080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:06:57.290040 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:35933 every 8 connection(s)
I20260620 08:06:57.294423 14122 log.cc:824] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:57.295089 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14080
I20260620 08:06:57.295377 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:46307
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:40507
--webserver_port=40443
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:57.299322 14122 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:06:57.299666 14122 tablet_bootstrap.cc:492] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap complete.
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:57.300925 14122 ts_tablet_manager.cc:1397] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20260620 08:06:57.302212 14122 raft_consensus.cc:348] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:57.302573 14122 raft_consensus.cc:729] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0250a0fd7c04c14888dfc80a5afc641, State: Initialized, Role: FOLLOWER
I20260620 08:06:57.302770 14122 consensus_queue.cc:260] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } } peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } }
I20260620 08:06:57.303143 14122 ts_tablet_manager.cc:1428] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260620 08:06:57.305603 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.291547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:51161 (local address 127.31.115.254:40507)
0620 08:06:57.291825 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.291837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.292106 (+ 269us) server_negotiation.cc:407] Connection header received
0620 08:06:57.293092 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.293099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.293161 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.293306 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:57.294364 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.295783 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.297439 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.298115 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.298236 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.301121 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:57.301127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:57.301127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:57.301142 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:57.303247 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.303797 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.303800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.303801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.303935 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.304258 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.304259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.304260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.304597 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:57.304689 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.305227 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.305408 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":142,"threads_started":1}
I20260620 08:06:57.305637 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.291393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:51161)
0620 08:06:57.291873 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.291965 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.292818 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.293391 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.293398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.293903 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:57.294210 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.294221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.295948 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.295949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.297293 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.297296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.298239 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.298241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.298399 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.299043 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:57.299055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:57.300969 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:57.303361 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:57.303367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:57.303370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:57.303702 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:57.304022 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:57.304023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:57.304024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:57.304184 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:57.304782 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:57.304791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:57.304990 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.305122 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.305337 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":112,"threads_started":1}
I20260620 08:06:57.307101 14230 heartbeater.cc:347] Connected to a master server at 127.31.115.254:40507
I20260620 08:06:57.307232 14230 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:57.307454 14230 heartbeater.cc:511] Master 127.31.115.254:40507 requested a full tablet report, sending...
I20260620 08:06:57.308367 14042 ts_manager.cc:194] Registered new tserver with Master: b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933)
I20260620 08:06:57.309718 14042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:51161
I20260620 08:06:57.310829 14230 heartbeater.cc:503] Master 127.31.115.254:40507 was elected leader, sending a full tablet report...
I20260620 08:06:57.317301 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.309079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35933 (local address 127.0.0.1:44276)
0620 08:06:57.309434 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.309479 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.309555 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.314007 (+ 4452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.314012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.314016 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:57.314101 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.314103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.315368 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.315370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.316201 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.316203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.316929 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.316930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.317025 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.317042 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.317144 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.317209 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":100,"threads_started":1}
I20260620 08:06:57.317301 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.309383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.31.115.193:35933)
0620 08:06:57.313764 (+ 4381us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.313770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.313785 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:06:57.313828 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.313830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.313849 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.313949 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:57.314196 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.315258 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.316314 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.316808 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.316924 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.316987 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.317165 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.317212 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4336,"thread_start_us":99,"threads_started":1}
I20260620 08:06:57.317968 14164 tablet_service.cc:1514] Processing DeleteTablet for tablet 8045de2fa5584ccea690bd96c98d9e75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:56 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:44276
I20260620 08:06:57.318621 14239 tablet_replica.cc:330] stopping tablet replica
I20260620 08:06:57.318754 14239 raft_consensus.cc:2229] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:06:57.318841 14239 raft_consensus.cc:2258] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:57.319336 14239 ts_tablet_manager.cc:1905] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:06:57.322042 14239 ts_tablet_manager.cc:1918] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:57.322139 14239 log.cc:1196] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/8045de2fa5584ccea690bd96c98d9e75
I20260620 08:06:57.322378 14239 ts_tablet_manager.cc:1939] T 8045de2fa5584ccea690bd96c98d9e75 P b0250a0fd7c04c14888dfc80a5afc641: Deleting consensus metadata
I20260620 08:06:57.323064 14029 catalog_manager.cc:4841] TS b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933): tablet 8045de2fa5584ccea690bd96c98d9e75 (table test-table [id=e92b909463204ce7a9f0375bde0babd7]) successfully deleted
W20260620 08:06:57.390154 14235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:57.390383 14235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:57.390408 14235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:57.391775 14235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:57.391848 14235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:57.391880 14235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:06:57.393445 14235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:46307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=40443
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:57.393775 14235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:57.394019 14235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:57.400032 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.400084 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.399998 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.400077 14235 server_base.cc:1034] running on GCE node
I20260620 08:06:57.400485 14235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:57.400784 14235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:57.402040 14235 hybrid_clock.cc:648] HybridClock initialized: now 1781942817402005 us; error 47 us; skew 500 ppm
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:57.404804 14235 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:06:57.405330 14235 webserver.cc:460] Webserver started at http://127.31.115.194:40443/ using document root <none> and password file <none>
I20260620 08:06:57.405575 14235 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:57.405704 14235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:57.407274 14235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:06:57.408047 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.408380 14235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:06:57.408540 14235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "6e3b9581f25544ea971223a561571e07"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "f68f61d350c39e6f580dfc8a464f4fe9"
server_key_iv: "3fb855bc162bbbed142f388c120c24fe"
server_key_version: "encryptionkey@0"
I20260620 08:06:57.408928 14235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:57.427270 14235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:57.429589 14235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:57.429849 14235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:57.430130 14235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:57.430465 14235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:57.430529 14235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.430575 14235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:57.430604 14235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.436947 14235 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:46307
I20260620 08:06:57.437000 14367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:46307 every 8 connection(s)
I20260620 08:06:57.437377 14235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:57.447230 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14235
I20260620 08:06:57.447510 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:44691
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:40507
--webserver_port=34473
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:57.447962 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.438232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:47603 (local address 127.31.115.254:40507)
0620 08:06:57.438298 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.438304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.438562 (+ 258us) server_negotiation.cc:407] Connection header received
0620 08:06:57.439042 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.439043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.439054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.439131 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:57.440124 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.440664 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.441740 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.441939 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.441998 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.444358 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:57.444359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:57.444359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:57.444366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:57.445955 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.446739 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.446741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.446741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.446803 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.447118 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.447119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.447119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.447268 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:57.447376 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.447773 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.447868 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:06:57.448163 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.438234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:47603)
0620 08:06:57.438400 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.438446 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.438933 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.439215 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.439219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.439790 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:57.439986 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.439991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.440759 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.440760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.441591 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.441594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.442031 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.442032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.442138 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.442652 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:57.442661 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:57.444267 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:57.446067 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:57.446075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:57.446077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:57.446644 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:57.446882 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:57.446883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:57.446883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:57.447000 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:57.447367 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:57.447375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:57.447530 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.447656 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.447859 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 08:06:57.448716 14368 heartbeater.cc:347] Connected to a master server at 127.31.115.254:40507
I20260620 08:06:57.448812 14368 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:57.449021 14368 heartbeater.cc:511] Master 127.31.115.254:40507 requested a full tablet report, sending...
I20260620 08:06:57.449456 14042 ts_manager.cc:194] Registered new tserver with Master: 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307)
I20260620 08:06:57.449879 14042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:47603
W20260620 08:06:57.532894 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:57.533202 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:57.533246 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:57.534672 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:57.534750 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:57.534798 14371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:06:57.536413 14371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:44691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=34473
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:57.536679 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:57.536948 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:06:57.542287 14371 server_base.cc:1034] running on GCE node
W20260620 08:06:57.542258 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.542244 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.542240 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.542689 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:57.542976 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:57.544163 14371 hybrid_clock.cc:648] HybridClock initialized: now 1781942817544131 us; error 53 us; skew 500 ppm
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:57.546931 14371 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:06:57.547400 14371 webserver.cc:460] Webserver started at http://127.31.115.195:34473/ using document root <none> and password file <none>
I20260620 08:06:57.547698 14371 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:57.547783 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:57.549252 14371 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:06:57.550028 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.550257 14371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:57.550370 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "b83a8abe1d7f401da80d6f2674abffa1"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "e90a1e9a95aaf35f5342bea6221cce61"
server_key_iv: "0cce43de13d42382af587642cb9838d1"
server_key_version: "encryptionkey@0"
I20260620 08:06:57.550761 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:57.567986 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:57.570405 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:57.570609 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:57.570856 14371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:57.571170 14371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:57.571223 14371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.571265 14371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:57.571306 14371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.577342 14371 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:44691
I20260620 08:06:57.577386 14499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:44691 every 8 connection(s)
I20260620 08:06:57.577819 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:06:57.583591 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14371
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:06:57.588177 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.578591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:33249 (local address 127.31.115.254:40507)
0620 08:06:57.578682 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.578685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.578882 (+ 197us) server_negotiation.cc:407] Connection header received
0620 08:06:57.579451 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.579453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.579465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.579519 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:57.580533 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.581045 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.582090 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.582301 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.582393 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.585039 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:57.585041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:57.585041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:57.585049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:57.586711 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.587157 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.587158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.587159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.587225 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.587502 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.587503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.587503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.587623 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:57.587698 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.587994 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.588080 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:06:57.588384 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.578549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:33249)
0620 08:06:57.578746 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.578791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.579321 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.579589 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.579594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.580180 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:57.580378 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.580382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.581149 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.581151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.581993 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.581995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.582405 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.582406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.582539 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.583135 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:57.583145 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:57.584936 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:57.586801 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:57.586812 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:57.586814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:57.587063 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:57.587299 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:57.587299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:57.587300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:57.587427 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:57.587701 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:57.587709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:57.587850 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.587958 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.588132 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260620 08:06:57.588951 14500 heartbeater.cc:347] Connected to a master server at 127.31.115.254:40507
I20260620 08:06:57.589069 14500 heartbeater.cc:464] Registering TS with master...
I20260620 08:06:57.589280 14500 heartbeater.cc:511] Master 127.31.115.254:40507 requested a full tablet report, sending...
I20260620 08:06:57.589762 14042 ts_manager.cc:194] Registered new tserver with Master: b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
I20260620 08:06:57.590135 14042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:33249
I20260620 08:06:57.591996 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.584105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.31.115.254:40507)
0620 08:06:57.584227 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.584229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.584346 (+ 117us) server_negotiation.cc:407] Connection header received
0620 08:06:57.584396 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.584397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.584415 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.584490 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:57.585238 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.585972 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.586867 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.587073 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.587161 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.588655 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:57.588657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:57.588657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:57.588665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:57.590418 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.591180 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.591182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.591182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.591242 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:57.591504 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:57.591504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:57.591505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:57.591619 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:57.591752 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.591846 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.591922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260620 08:06:57.593004 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:06:57.596098 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.593462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.31.115.254:40507)
0620 08:06:57.593502 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.593509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.593588 (+ 79us) server_negotiation.cc:407] Connection header received
0620 08:06:57.593700 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.593700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.593728 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.593789 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:06:57.593916 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.594377 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.595195 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.595393 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.595568 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.595743 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:06:57.595858 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:06:57.595945 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.596024 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.596047 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:06:57.596392 14042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:06:57.598104 14042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:06:57.600610 14164 tablet_service.cc:1467] Processing CreateTablet for tablet 8169c53b2cb3480dac47f5b3edfcd5d8 (DEFAULT_TABLE table=test-table [id=ab03b4e20ede479b904f37f78aa0e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:57.600823 14164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8169c53b2cb3480dac47f5b3edfcd5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:57.602500 14122 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap starting.
I20260620 08:06:57.602880 14122 tablet_bootstrap.cc:654] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:57.603516 14122 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: No bootstrap required, opened a new log
I20260620 08:06:57.603569 14122 ts_tablet_manager.cc:1397] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:57.603688 14122 raft_consensus.cc:348] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.603762 14122 raft_consensus.cc:374] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:57.603830 14122 raft_consensus.cc:729] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0250a0fd7c04c14888dfc80a5afc641, State: Initialized, Role: FOLLOWER
I20260620 08:06:57.603878 14122 consensus_queue.cc:260] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.603912 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.600147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:46307 (local address 127.0.0.1:37408)
0620 08:06:57.600252 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.600264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.600297 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.600654 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.600656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.600658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:57.600706 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.600708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.601816 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.601818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.602598 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.602601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.603666 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.603667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.603728 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.603733 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.603790 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.603836 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 08:06:57.603968 14122 ts_tablet_manager.cc:1428] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:06:57.603968 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.600261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.31.115.194:46307)
0620 08:06:57.600476 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.600482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.600494 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:06:57.600539 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.600541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.600558 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.600645 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:57.600762 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.601712 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.602709 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.603482 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.603577 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.603646 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.603831 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.603883 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
I20260620 08:06:57.604666 14302 tablet_service.cc:1467] Processing CreateTablet for tablet 8169c53b2cb3480dac47f5b3edfcd5d8 (DEFAULT_TABLE table=test-table [id=ab03b4e20ede479b904f37f78aa0e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:57.604835 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.600193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44691 (local address 127.0.0.1:33402)
0620 08:06:57.600522 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.600532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.600592 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.600979 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.600980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.600982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:57.601018 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.601019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.602347 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.602348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.603524 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.603527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.604582 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.604583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.604648 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.604652 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.604730 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.604770 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":59,"threads_started":1}
I20260620 08:06:57.604871 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.600308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.31.115.195:44691)
0620 08:06:57.600504 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.600552 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.600585 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:06:57.600811 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.600814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.600831 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.600900 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:57.601109 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.602228 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.603643 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.604413 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.604493 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.604544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.604748 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.604792 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260620 08:06:57.605139 14302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8169c53b2cb3480dac47f5b3edfcd5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:57.605598 14434 tablet_service.cc:1467] Processing CreateTablet for tablet 8169c53b2cb3480dac47f5b3edfcd5d8 (DEFAULT_TABLE table=test-table [id=ab03b4e20ede479b904f37f78aa0e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:06:57.605909 14434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8169c53b2cb3480dac47f5b3edfcd5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:06:57.608292 14510 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Bootstrap starting.
I20260620 08:06:57.608589 14511 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Bootstrap starting.
I20260620 08:06:57.608950 14510 tablet_bootstrap.cc:654] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:57.609164 14511 tablet_bootstrap.cc:654] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Neither blocks nor log segments found. Creating new log.
I20260620 08:06:57.609494 14510 log.cc:824] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:57.609691 14511 log.cc:824] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:57.610387 14510 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: No bootstrap required, opened a new log
I20260620 08:06:57.610447 14510 ts_tablet_manager.cc:1397] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:06:57.611758 14510 raft_consensus.cc:348] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.611902 14510 raft_consensus.cc:374] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:57.611984 14510 raft_consensus.cc:729] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3b9581f25544ea971223a561571e07, State: Initialized, Role: FOLLOWER
I20260620 08:06:57.612108 14510 consensus_queue.cc:260] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.612341 14510 ts_tablet_manager.cc:1428] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:06:57.612411 14368 heartbeater.cc:503] Master 127.31.115.254:40507 was elected leader, sending a full tablet report...
I20260620 08:06:57.613036 14511 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: No bootstrap required, opened a new log
I20260620 08:06:57.613131 14511 ts_tablet_manager.cc:1397] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 08:06:57.614732 14511 raft_consensus.cc:348] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.614866 14511 raft_consensus.cc:374] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:06:57.614888 14511 raft_consensus.cc:729] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83a8abe1d7f401da80d6f2674abffa1, State: Initialized, Role: FOLLOWER
I20260620 08:06:57.615016 14511 consensus_queue.cc:260] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.615300 14511 ts_tablet_manager.cc:1428] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 08:06:57.615334 14500 heartbeater.cc:503] Master 127.31.115.254:40507 was elected leader, sending a full tablet report...
W20260620 08:06:57.688058 14369 tablet.cc:2369] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:06:57.732864 14514 raft_consensus.cc:482] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:06:57.733075 14514 raft_consensus.cc:504] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.733527 14514 leader_election.cc:288] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691), b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933)
I20260620 08:06:57.737067 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.733869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:54701 (local address 127.31.115.193:35933)
0620 08:06:57.733918 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.733920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.734132 (+ 212us) server_negotiation.cc:407] Connection header received
0620 08:06:57.734240 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.734242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.734251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.734344 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:57.734658 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.735206 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.736290 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.736687 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.736776 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.736788 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.736948 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.736995 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:06:57.737079 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.733787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35933 (local address 127.31.115.194:54701)
0620 08:06:57.734033 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.734058 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.734150 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.734370 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.734371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.734372 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:57.734555 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.734558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.735325 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.735326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.736187 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.736189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.736785 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.736787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.736869 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.736873 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.736941 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.736984 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":79,"thread_start_us":119,"threads_started":1}
I20260620 08:06:57.737279 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.733787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44691 (local address 127.31.115.194:34989)
0620 08:06:57.733894 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.733908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:57.733968 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:57.734107 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:57.734108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:57.734109 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:06:57.734206 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.734209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.735389 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.735390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.736201 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:57.736204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.736956 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.736958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:57.737054 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.737057 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:06:57.737134 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:06:57.737185 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 08:06:57.737455 14184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8169c53b2cb3480dac47f5b3edfcd5d8" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0250a0fd7c04c14888dfc80a5afc641" is_pre_election: true
I20260620 08:06:57.737694 14184 raft_consensus.cc:2454] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 0.
I20260620 08:06:57.737903 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.733877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:34989 (local address 127.31.115.195:44691)
0620 08:06:57.733918 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.733920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.733955 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:06:57.734037 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.734039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.734049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.734103 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:06:57.734302 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.735030 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.736306 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.736848 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.736967 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.736976 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.737141 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.737306 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:06:57.737998 14258 leader_election.cc:302] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b9581f25544ea971223a561571e07, b0250a0fd7c04c14888dfc80a5afc641; no voters:
I20260620 08:06:57.738159 14514 raft_consensus.cc:2781] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:06:57.738214 14514 raft_consensus.cc:482] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:06:57.738237 14514 raft_consensus.cc:3037] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:57.738265 14454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8169c53b2cb3480dac47f5b3edfcd5d8" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83a8abe1d7f401da80d6f2674abffa1" is_pre_election: true
I20260620 08:06:57.738461 14454 raft_consensus.cc:2454] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 0.
I20260620 08:06:57.739141 14514 raft_consensus.cc:504] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.739308 14514 leader_election.cc:288] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 election: Requested vote from peers b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691), b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933)
I20260620 08:06:57.739460 14454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8169c53b2cb3480dac47f5b3edfcd5d8" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83a8abe1d7f401da80d6f2674abffa1"
I20260620 08:06:57.739538 14454 raft_consensus.cc:3037] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:57.739593 14184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8169c53b2cb3480dac47f5b3edfcd5d8" candidate_uuid: "6e3b9581f25544ea971223a561571e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0250a0fd7c04c14888dfc80a5afc641"
I20260620 08:06:57.739681 14184 raft_consensus.cc:3037] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:06:57.740552 14454 raft_consensus.cc:2454] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 1.
I20260620 08:06:57.740808 14256 leader_election.cc:302] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b9581f25544ea971223a561571e07, b83a8abe1d7f401da80d6f2674abffa1; no voters:
I20260620 08:06:57.740916 14514 raft_consensus.cc:2781] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:06:57.741012 14184 raft_consensus.cc:2454] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3b9581f25544ea971223a561571e07 in term 1.
I20260620 08:06:57.741072 14514 raft_consensus.cc:686] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 1 LEADER]: Becoming Leader. State: Replica: 6e3b9581f25544ea971223a561571e07, State: Running, Role: LEADER
I20260620 08:06:57.741194 14514 consensus_queue.cc:237] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:57.742138 14042 catalog_manager.cc:5495] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3b9581f25544ea971223a561571e07 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "6e3b9581f25544ea971223a561571e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20260620 08:06:57.753139 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.750450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.31.115.194:46307)
0620 08:06:57.750534 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:57.750536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:57.750552 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:06:57.750591 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:57.750592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:57.750611 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:57.750678 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:06:57.750831 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.751332 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.752102 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:57.752300 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:57.752367 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:57.752590 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:06:57.752848 (+ 258us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:06:57.752946 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:57.753000 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 08:06:57.753023 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:06:57.755563 14454 raft_consensus.cc:1261] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Refusing update from remote peer 6e3b9581f25544ea971223a561571e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:57.755650 14184 raft_consensus.cc:1261] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Refusing update from remote peer 6e3b9581f25544ea971223a561571e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:06:57.755962 14514 consensus_queue.cc:1035] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:57.756127 14517 consensus_queue.cc:1035] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:06:57.758349 14519 mvcc.cc:205] Tried to move back new op lower bound from 7298837781523378176 to 7298837781468454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298837781523378176}]
I20260620 08:06:57.759231 14521 mvcc.cc:205] Tried to move back new op lower bound from 7298837781523378176 to 7298837781468454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298837781523378176}]
I20260620 08:06:57.759609 14520 mvcc.cc:205] Tried to move back new op lower bound from 7298837781523378176 to 7298837781468454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298837781523378176 or (T in {7298837781523378176})}]
I20260620 08:06:57.760999 14041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:06:57.761108 14041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:06:57.762081 14041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Sending DeleteTablet for 3 replicas of tablet 8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:06:57.762466 14302 tablet_service.cc:1514] Processing DeleteTablet for tablet 8169c53b2cb3480dac47f5b3edfcd5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:57 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:37408
I20260620 08:06:57.762457 14164 tablet_service.cc:1514] Processing DeleteTablet for tablet 8169c53b2cb3480dac47f5b3edfcd5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:57 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:44276
I20260620 08:06:57.762578 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14080
I20260620 08:06:57.762676 14239 tablet_replica.cc:330] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: stopping tablet replica
I20260620 08:06:57.762739 14239 raft_consensus.cc:2229] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:06:57.762785 14239 raft_consensus.cc:2258] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:57.763260 14529 tablet_replica.cc:330] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: stopping tablet replica
I20260620 08:06:57.763374 14529 raft_consensus.cc:2229] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:06:57.763480 14529 raft_consensus.cc:2258] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:57.763909 14529 ts_tablet_manager.cc:1905] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:06:57.765420 14529 ts_tablet_manager.cc:1918] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:57.765553 14529 log.cc:1196] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:06:57.765965 14529 ts_tablet_manager.cc:1939] T 8169c53b2cb3480dac47f5b3edfcd5d8 P 6e3b9581f25544ea971223a561571e07: Deleting consensus metadata
I20260620 08:06:57.766461 14029 catalog_manager.cc:4841] TS 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307): tablet 8169c53b2cb3480dac47f5b3edfcd5d8 (table test-table [id=ab03b4e20ede479b904f37f78aa0e03c]) successfully deleted
I20260620 08:06:57.766484 14434 tablet_service.cc:1514] Processing DeleteTablet for tablet 8169c53b2cb3480dac47f5b3edfcd5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:06:57 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:33402
I20260620 08:06:57.766783 14530 tablet_replica.cc:330] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: stopping tablet replica
I20260620 08:06:57.766920 14530 raft_consensus.cc:2229] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:06:57.767060 14530 raft_consensus.cc:2258] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:06:57.767479 14530 ts_tablet_manager.cc:1905] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:06:57.768816 14530 ts_tablet_manager.cc:1918] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:06:57.768910 14530 log.cc:1196] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:06:57.769176 14530 ts_tablet_manager.cc:1939] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b83a8abe1d7f401da80d6f2674abffa1: Deleting consensus metadata
I20260620 08:06:57.769641 14027 catalog_manager.cc:4841] TS b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691): tablet 8169c53b2cb3480dac47f5b3edfcd5d8 (table test-table [id=ab03b4e20ede479b904f37f78aa0e03c]) successfully deleted
W20260620 08:06:57.771698 14029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:35933 (error 108)
I20260620 08:06:57.771905 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14235
I20260620 08:06:57.773137 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:57.772991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:35933 (local address 127.0.0.1:44278)
0620 08:06:57.773055 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:57.773103 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:35933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 08:06:57.773357 14029 catalog_manager.cc:4584] TS b0250a0fd7c04c14888dfc80a5afc641 (127.31.115.193:35933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8169c53b2cb3480dac47f5b3edfcd5d8: Network error: Client connection negotiation failed: client connection to 127.31.115.193:35933: connect: Connection refused (error 111)
I20260620 08:06:57.776918 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14371
I20260620 08:06:57.781575 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14009
I20260620 08:06:57.787452 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40507
--webserver_interface=127.31.115.254
--webserver_port=37155
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:06:57.875993 14532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:57.876221 14532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:57.876245 14532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:57.877456 14532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:06:57.877512 14532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:57.877527 14532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:57.877540 14532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:06:57.877552 14532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:06:57.879201 14532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=37155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:57.879498 14532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:57.879756 14532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:57.884898 14538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:57.884907 14540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.885237 14532 server_base.cc:1034] running on GCE node
W20260620 08:06:57.884992 14537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:57.885596 14532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:57.885967 14532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:57.887195 14532 hybrid_clock.cc:648] HybridClock initialized: now 1781942817887177 us; error 26 us; skew 500 ppm
Jun 20 08:06:57 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:57.890113 14532 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:06:57.890614 14532 webserver.cc:460] Webserver started at http://127.31.115.254:37155/ using document root <none> and password file <none>
I20260620 08:06:57.890887 14532 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:57.890939 14532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:57.892552 14532 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 08:06:57.893959 14547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:57.894203 14532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:57.894315 14532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "43b1c329edfd4379ba3052ab534400e2"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "adc714d666c63f3d24b36b9a0fb0bb13"
server_key_iv: "fca6c28ce58a0b25727c9503954719f5"
server_key_version: "encryptionkey@0"
I20260620 08:06:57.894807 14532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:57.915226 14532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:57.917560 14532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:57.917793 14532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:57.922214 14532 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:40507
I20260620 08:06:57.922308 14599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:40507 every 8 connection(s)
I20260620 08:06:57.922806 14532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:06:57.923282 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14532
I20260620 08:06:57.923599 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:35933
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:40507
--webserver_port=46003
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:57.925064 14600 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:06:57.925966 14600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap starting.
I20260620 08:06:57.935415 14600 log.cc:824] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:57.937947 14600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:06:57.938287 14600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2: Bootstrap complete.
I20260620 08:06:57.940639 14600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.940908 14600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b1c329edfd4379ba3052ab534400e2, State: Initialized, Role: FOLLOWER
I20260620 08:06:57.941130 14600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.941231 14600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:06:57.941286 14600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:06:57.941335 14600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 08:06:57.942529 14600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.942659 14600 leader_election.cc:302] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b1c329edfd4379ba3052ab534400e2; no voters:
I20260620 08:06:57.942854 14600 leader_election.cc:288] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 08:06:57.942945 14604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 08:06:57.943182 14600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:06:57.943197 14604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [term 3 LEADER]: Becoming Leader. State: Replica: 43b1c329edfd4379ba3052ab534400e2, State: Running, Role: LEADER
I20260620 08:06:57.943353 14604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } }
I20260620 08:06:57.943826 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b1c329edfd4379ba3052ab534400e2. Latest consensus state: current_term: 3 leader_uuid: "43b1c329edfd4379ba3052ab534400e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } } }
I20260620 08:06:57.943991 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:57.944195 14605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "43b1c329edfd4379ba3052ab534400e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b1c329edfd4379ba3052ab534400e2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40507 } } }
I20260620 08:06:57.944268 14615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:06:57.944350 14605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b1c329edfd4379ba3052ab534400e2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:06:57.945267 14615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:06:57.945542 14615 catalog_manager.cc:1259] Loaded cluster ID: 481f982a25fb4a52828ad1506906f614
I20260620 08:06:57.945633 14615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 08:06:57.945969 14615 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc1364009ad google::LogMessage::Fail() at ??:0
@ 0x7fc13640388a google::LogMessage::SendToLog() at ??:0
@ 0x7fc1364004de google::LogMessage::Flush() at ??:0
@ 0x7fc136401a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc13e3210f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc13e32a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc1382ce0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc1382c3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc1391276db start_thread at ??:0
@ 0x7fc134efe61f clone at ??:0
W20260620 08:06:58.026692 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:58.026942 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:58.026980 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:58.028343 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:58.028411 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:58.028472 14602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:06:58.030187 14602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:35933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=46003
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:58.030478 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:58.030730 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:58.036188 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:58.036207 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:58.036260 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:58.036679 14602 server_base.cc:1034] running on GCE node
I20260620 08:06:58.036922 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:58.037184 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:58.038400 14602 hybrid_clock.cc:648] HybridClock initialized: now 1781942818038382 us; error 33 us; skew 500 ppm
Jun 20 08:06:58 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:58.041198 14602 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:06:58.041831 14602 webserver.cc:460] Webserver started at http://127.31.115.193:46003/ using document root <none> and password file <none>
I20260620 08:06:58.042083 14602 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:58.042153 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:58.043660 14602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:58.044486 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.044723 14602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:58.044855 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "b0250a0fd7c04c14888dfc80a5afc641"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "d581f8ea5d826a80b1f84972d8288422"
server_key_iv: "e369e2c28f60ac52c727a036544085b3"
server_key_version: "encryptionkey@0"
I20260620 08:06:58.045274 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:58.080646 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:58.083257 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:58.083441 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:58.083706 14602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:58.084189 14653 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:06:58.086376 14602 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:06:58.086609 14602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 08:06:58.086692 14602 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:06:58.087373 14602 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:06:58.087431 14602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:06:58.087527 14653 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap starting.
I20260620 08:06:58.095615 14602 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:35933
I20260620 08:06:58.096032 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:06:58.096664 14760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:35933 every 8 connection(s)
I20260620 08:06:58.099480 14653 log.cc:824] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Log is configured to *not* fsync() on all Append() calls
I20260620 08:06:58.099918 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14602
I20260620 08:06:58.100306 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:46307
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:40507
--webserver_port=40443
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:58.102924 14653 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:06:58.103276 14653 tablet_bootstrap.cc:492] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Bootstrap complete.
I20260620 08:06:58.104290 14653 ts_tablet_manager.cc:1397] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20260620 08:06:58.105484 14653 raft_consensus.cc:348] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:58.105808 14653 raft_consensus.cc:729] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0250a0fd7c04c14888dfc80a5afc641, State: Initialized, Role: FOLLOWER
I20260620 08:06:58.105967 14653 consensus_queue.cc:260] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:58.106290 14653 ts_tablet_manager.cc:1428] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260620 08:06:58.173502 14763 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:06:58.096788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:38315)
0620 08:06:58.096983 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.097029 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:58.097527 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:58.173279 (+ 75752us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
W20260620 08:06:58.173842 14761 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:40507 (0 consecutive failures): Network error: Failed to ping master at 127.31.115.254:40507: Client connection negotiation failed: client connection to 127.31.115.254:40507: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:06:58.174219 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.174084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:39869)
0620 08:06:58.174147 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.174199 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 08:06:58.174587 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.174482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:54689)
0620 08:06:58.174536 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.174562 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:06:58.174711 14761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:06:58.203734 14765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:58.204047 14765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:58.204097 14765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:58.205585 14765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:58.205717 14765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:58.205775 14765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:06:58.207734 14765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:46307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=40443
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:58.208082 14765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:58.208366 14765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:58.215214 14771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:58.215224 14774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:58.215233 14772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:58.216288 14765 server_base.cc:1034] running on GCE node
I20260620 08:06:58.216557 14765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:58.216852 14765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:58.218022 14765 hybrid_clock.cc:648] HybridClock initialized: now 1781942818218009 us; error 33 us; skew 500 ppm
Jun 20 08:06:58 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:58.220844 14765 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:06:58.221344 14765 webserver.cc:460] Webserver started at http://127.31.115.194:40443/ using document root <none> and password file <none>
I20260620 08:06:58.221642 14765 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:58.221704 14765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:58.223212 14765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:58.224130 14781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.224524 14765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:58.224679 14765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "6e3b9581f25544ea971223a561571e07"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "f68f61d350c39e6f580dfc8a464f4fe9"
server_key_iv: "3fb855bc162bbbed142f388c120c24fe"
server_key_version: "encryptionkey@0"
I20260620 08:06:58.225083 14765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:58.233538 14765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:58.236091 14765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:58.236267 14765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:58.236527 14765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:58.236852 14765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:58.236908 14765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.236951 14765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:58.236996 14765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.245047 14765 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:46307
I20260620 08:06:58.245136 14894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:46307 every 8 connection(s)
I20260620 08:06:58.245523 14765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:06:58.247058 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.246305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:36475)
0620 08:06:58.246652 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.246823 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260620 08:06:58.247545 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14765
W20260620 08:06:58.247864 14895 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:40507 (0 consecutive failures): Network error: Failed to ping master at 127.31.115.254:40507: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
I20260620 08:06:58.247907 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:44691
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:40507
--webserver_port=34473
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:06:58.248281 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.248162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:56825)
0620 08:06:58.248210 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.248249 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 08:06:58.248637 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.248539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:57915)
0620 08:06:58.248589 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.248613 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:06:58.248792 14895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:06:58.335132 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:06:58.335359 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:06:58.335386 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:06:58.336599 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:06:58.336650 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:06:58.336679 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:06:58.338255 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:44691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=34473
--tserver_master_addrs=127.31.115.254:40507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:06:58.338519 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:06:58.338783 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:06:58.344257 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:58.344208 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:06:58.344208 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:06:58.344460 14898 server_base.cc:1034] running on GCE node
I20260620 08:06:58.344686 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:06:58.344964 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:06:58.346133 14898 hybrid_clock.cc:648] HybridClock initialized: now 1781942818346126 us; error 48 us; skew 500 ppm
Jun 20 08:06:58 dist-test-slave-c4w8 krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:06:58.349188 14898 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:06:58.349797 14898 webserver.cc:460] Webserver started at http://127.31.115.195:34473/ using document root <none> and password file <none>
I20260620 08:06:58.350000 14898 fs_manager.cc:359] Metadata directory not provided
I20260620 08:06:58.350046 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:06:58.351362 14898 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:58.352026 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.352249 14898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:06:58.352326 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "b83a8abe1d7f401da80d6f2674abffa1"
format_stamp: "Formatted at 2026-06-20 08:06:56 on dist-test-slave-c4w8"
server_key: "e90a1e9a95aaf35f5342bea6221cce61"
server_key_iv: "0cce43de13d42382af587642cb9838d1"
server_key_version: "encryptionkey@0"
I20260620 08:06:58.352734 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:06:58.373736 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:06:58.376791 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:06:58.376932 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:06:58.377269 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:06:58.377650 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:06:58.377712 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.377748 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:06:58.377777 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:06:58.384421 14898 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:44691
I20260620 08:06:58.384464 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:44691 every 8 connection(s)
I20260620 08:06:58.384864 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:06:58.386083 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.385642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:39009)
0620 08:06:58.385834 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.385923 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
W20260620 08:06:58.386565 15027 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:40507 (0 consecutive failures): Network error: Failed to ping master at 127.31.115.254:40507: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
I20260620 08:06:58.386899 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.386793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:46499)
0620 08:06:58.386851 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.386880 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:06:58.387387 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:58.387266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:48531)
0620 08:06:58.387338 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:58.387366 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 08:06:58.387584 15027 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:06:58.394734 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 14898
I20260620 08:06:59.175403 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.175124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:44365)
0620 08:06:59.175282 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:59.175319 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260620 08:06:59.249497 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.249201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:53103)
0620 08:06:59.249382 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:59.249428 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260620 08:06:59.388290 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.388022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:56409)
0620 08:06:59.388184 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:59.388231 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260620 08:06:59.755221 15034 raft_consensus.cc:482] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:06:59.755347 15034 raft_consensus.cc:504] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:06:59.755753 15034 leader_election.cc:288] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307), b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
Jun 20 08:06:59 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.194@KRBTEST.COM
Jun 20 08:06:59 dist-test-slave-c4w8 krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.195@KRBTEST.COM
I20260620 08:06:59.767896 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.756021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:44691 (local address 127.31.115.193:53117)
0620 08:06:59.756208 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:59.756220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:59.756278 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:59.757288 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:59.757289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:59.757763 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:59.757925 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:59.757932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.759245 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.759246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:59.760055 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:59.760058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.760687 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.760688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:59.760812 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:59.761428 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:59.761438 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:59.763703 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:59.765870 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:59.765871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:59.765871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:59.766125 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:59.766547 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:59.766548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:59.766548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:59.766718 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:59.767244 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:59.767252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:59.767465 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 08:06:59.767549 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 08:06:59.767771 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":14,"thread_start_us":63,"threads_started":1}
I20260620 08:06:59.767927 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.756061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:56333 (local address 127.31.115.194:46307)
0620 08:06:59.756295 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:59.756304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:59.756324 (+ 20us) server_negotiation.cc:407] Connection header received
0620 08:06:59.756846 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:59.756848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:59.756894 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:59.757017 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:59.758107 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.759082 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.760173 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.760589 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.760732 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:59.763441 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:59.763447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:59.763447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:59.763462 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:59.765542 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:59.766206 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:59.766210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:59.766211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:59.766348 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:59.766801 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:59.766802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:59.766803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:59.767166 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:59.767257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:59.767662 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 08:06:59.767804 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":60,"threads_started":1}
I20260620 08:06:59.767927 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.756090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:53117 (local address 127.31.115.195:44691)
0620 08:06:59.756288 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 08:06:59.756300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:06:59.756335 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:06:59.757089 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:06:59.757093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:06:59.757147 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:06:59.757289 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:06:59.758198 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.759080 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.760171 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.760583 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.760718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:59.763854 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:06:59.763858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:06:59.763859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:06:59.763878 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:06:59.765792 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:59.766206 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:59.766210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:59.766211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:59.766353 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:06:59.766801 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:06:59.766802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:06:59.766803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:06:59.767166 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:06:59.767257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:06:59.767662 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 08:06:59.767804 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260620 08:06:59.767925 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:06:59.755938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:46307 (local address 127.31.115.193:56333)
0620 08:06:59.756071 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:06:59.756084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:06:59.756144 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:06:59.757107 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:06:59.757111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:06:59.757725 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:06:59.757925 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:59.757933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.759213 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.759215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:59.760055 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:06:59.760058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:06:59.760699 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:06:59.760700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:06:59.760818 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:06:59.761494 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:06:59.761499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:06:59.763227 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:06:59.765681 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:59.765692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:59.765695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:59.766095 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:59.766547 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:06:59.766548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:06:59.766548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:06:59.766718 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:06:59.767244 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:06:59.767252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:06:59.767421 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 08:06:59.767549 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 08:06:59.767826 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":216,"thread_start_us":61,"threads_started":1}
W20260620 08:06:59.768950 14648 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
W20260620 08:06:59.769285 14650 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:06:59.769356 14650 leader_election.cc:302] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0250a0fd7c04c14888dfc80a5afc641; no voters: 6e3b9581f25544ea971223a561571e07, b83a8abe1d7f401da80d6f2674abffa1
I20260620 08:06:59.769570 15034 raft_consensus.cc:2726] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:07:00.176225 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:00.176068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:54773)
0620 08:07:00.176141 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:00.176184 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 08:07:00.250449 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:00.250135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:36881)
0620 08:07:00.250286 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:00.250389 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260620 08:07:00.389163 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:00.388896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:57797)
0620 08:07:00.389063 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:00.389110 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260620 08:07:01.177315 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:01.176949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:54297)
0620 08:07:01.177169 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:01.177238 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260620 08:07:01.251616 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:01.251344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:36245)
0620 08:07:01.251519 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:01.251566 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260620 08:07:01.390328 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:01.390007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:52043)
0620 08:07:01.390217 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:01.390264 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260620 08:07:01.391995 15044 raft_consensus.cc:482] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:07:01.392109 15044 raft_consensus.cc:504] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:07:01.392336 15044 leader_election.cc:288] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307), b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
W20260620 08:07:01.392897 14648 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
W20260620 08:07:01.393020 14650 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:07:01.393133 14650 leader_election.cc:302] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0250a0fd7c04c14888dfc80a5afc641; no voters: 6e3b9581f25544ea971223a561571e07, b83a8abe1d7f401da80d6f2674abffa1
I20260620 08:07:01.393232 15044 raft_consensus.cc:2726] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:07:02.178719 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:02.178396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:47431)
0620 08:07:02.178615 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:02.178668 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260620 08:07:02.252720 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:02.252456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:58063)
0620 08:07:02.252629 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:02.252673 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260620 08:07:02.391464 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:02.391222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:36113)
0620 08:07:02.391369 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:02.391413 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260620 08:07:03.179730 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:03.179440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:42337)
0620 08:07:03.179626 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:03.179676 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260620 08:07:03.253684 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:03.253348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:49127)
0620 08:07:03.253523 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:03.253570 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260620 08:07:03.392433 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:03.392126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:53489)
0620 08:07:03.392282 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:03.392375 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260620 08:07:04.180843 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:04.180537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:47527)
0620 08:07:04.180719 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:04.180753 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260620 08:07:04.254510 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:04.254250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:38405)
0620 08:07:04.254414 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:04.254461 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260620 08:07:04.393639 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:04.393253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:42423)
0620 08:07:04.393491 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:04.393551 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":125,"threads_started":1}
I20260620 08:07:04.533295 15054 raft_consensus.cc:482] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:07:04.533432 15054 raft_consensus.cc:504] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:07:04.533681 15054 leader_election.cc:288] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307), b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
W20260620 08:07:04.534093 14650 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
W20260620 08:07:04.534202 14648 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:07:04.534276 14648 leader_election.cc:302] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0250a0fd7c04c14888dfc80a5afc641; no voters: 6e3b9581f25544ea971223a561571e07, b83a8abe1d7f401da80d6f2674abffa1
I20260620 08:07:04.534350 15054 raft_consensus.cc:2726] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:07:05.181820 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:05.181457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:34587)
0620 08:07:05.181722 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:05.181769 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":73,"threads_started":1}
I20260620 08:07:05.255435 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:05.255189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:38299)
0620 08:07:05.255331 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:05.255384 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260620 08:07:05.394443 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:05.394178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:43679)
0620 08:07:05.394340 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:05.394386 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260620 08:07:06.182777 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:06.182459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:49769)
0620 08:07:06.182668 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:06.182718 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260620 08:07:06.256330 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:06.256059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:35797)
0620 08:07:06.256235 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:06.256281 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260620 08:07:06.395385 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:06.395092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:44631)
0620 08:07:06.395285 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:06.395332 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":112,"threads_started":1}
I20260620 08:07:07.183820 15061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:07.183512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:38097)
0620 08:07:07.183721 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:07.183769 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260620 08:07:07.257213 15062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:07.256940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:54773)
0620 08:07:07.257118 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:07.257163 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260620 08:07:07.396363 15063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:07.396033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:42587)
0620 08:07:07.396244 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:07.396305 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260620 08:07:08.184824 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:08.184474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:43173)
0620 08:07:08.184665 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:08.184711 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260620 08:07:08.201066 15065 raft_consensus.cc:482] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:07:08.201175 15065 raft_consensus.cc:504] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b9581f25544ea971223a561571e07" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 46307 } } peers { permanent_uuid: "b83a8abe1d7f401da80d6f2674abffa1" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 44691 } } peers { permanent_uuid: "b0250a0fd7c04c14888dfc80a5afc641" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 35933 } }
I20260620 08:07:08.201378 15065 leader_election.cc:288] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307), b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691)
W20260620 08:07:08.201872 14650 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e3b9581f25544ea971223a561571e07 (127.31.115.194:46307): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
W20260620 08:07:08.201948 14648 leader_election.cc:341] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b83a8abe1d7f401da80d6f2674abffa1 (127.31.115.195:44691): Not found: Tablet not found: 8169c53b2cb3480dac47f5b3edfcd5d8
I20260620 08:07:08.201980 14648 leader_election.cc:302] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0250a0fd7c04c14888dfc80a5afc641; no voters: 6e3b9581f25544ea971223a561571e07, b83a8abe1d7f401da80d6f2674abffa1
I20260620 08:07:08.202068 15065 raft_consensus.cc:2726] T 8169c53b2cb3480dac47f5b3edfcd5d8 P b0250a0fd7c04c14888dfc80a5afc641 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:07:08.258033 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:08.257772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:40667)
0620 08:07:08.257914 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:08.257952 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260620 08:07:08.397439 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:08.397127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:52085)
0620 08:07:08.397335 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:08.397381 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260620 08:07:09.185868 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:09.185557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:46721)
0620 08:07:09.185772 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:09.185819 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":133,"threads_started":1}
I20260620 08:07:09.258991 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:09.258734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:43989)
0620 08:07:09.258882 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:09.258941 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260620 08:07:09.398378 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:09.398118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:38965)
0620 08:07:09.398261 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:09.398319 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260620 08:07:10.186682 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:10.186429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:38059)
0620 08:07:10.186589 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:10.186632 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260620 08:07:10.259850 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:10.259592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:37575)
0620 08:07:10.259750 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:10.259799 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260620 08:07:10.399156 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:10.398913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:56801)
0620 08:07:10.399059 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:10.399107 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260620 08:07:11.187649 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:11.187390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:43753)
0620 08:07:11.187556 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:11.187600 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260620 08:07:11.260699 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:11.260407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:57475)
0620 08:07:11.260598 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:11.260640 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260620 08:07:11.399901 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:11.399679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:35175)
0620 08:07:11.399812 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:11.399851 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260620 08:07:12.188571 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:12.188275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:55423)
0620 08:07:12.188455 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:12.188515 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260620 08:07:12.261485 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:12.261228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:38855)
0620 08:07:12.261367 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:12.261413 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260620 08:07:12.400795 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:12.400502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:60261)
0620 08:07:12.400686 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:12.400738 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260620 08:07:13.189502 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:13.189243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.193:45583)
0620 08:07:13.189408 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:13.189453 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260620 08:07:13.262274 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:13.262040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.194:37703)
0620 08:07:13.262179 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:13.262220 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260620 08:07:13.401823 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:13.401523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40507 (local address 127.31.115.195:53689)
0620 08:07:13.401722 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:13.401772 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260620 08:07:13.406009 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14602
I20260620 08:07:13.411593 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14765
I20260620 08:07:13.416652 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 14898
2026-06-20T08:07:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19362 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:07:14 dist-test-slave-c4w8 krb5kdc[15085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:07:14 dist-test-slave-c4w8 krb5kdc[15085](info): set up 2 sockets
Jun 20 08:07:14 dist-test-slave-c4w8 krb5kdc[15085](info): commencing operation
krb5kdc: starting...
W20260620 08:07:15.410225 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:07:15 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:07:15.545960 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36017
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:15.634366 15098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:15.634645 15098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:15.634688 15098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:15.635918 15098 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:07:15.635999 15098 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:07:15.636021 15098 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:07:15.636041 15098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:07:15.636075 15098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:15.636104 15098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:15.636134 15098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:07:15.636161 15098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:07:15.637773 15098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:15.638091 15098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:15.638336 15098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:15.644018 15106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:15.644057 15103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:15.644096 15104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:15.644116 15098 server_base.cc:1034] running on GCE node
I20260620 08:07:15.644538 15098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:15.644860 15098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:15.646039 15098 hybrid_clock.cc:648] HybridClock initialized: now 1781942835646004 us; error 45 us; skew 500 ppm
Jun 20 08:07:15 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942835, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:15.648877 15098 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:07:15.649451 15098 webserver.cc:460] Webserver started at http://127.31.115.254:38925/ using document root <none> and password file <none>
I20260620 08:07:15.649801 15098 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:15.649919 15098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:15.650125 15098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:07:15.651329 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "0858f2fd1499472d98b92560f321f549"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "ab360d9dac96ba40ad6954fe12f48fac"
server_key_iv: "51d31b22099f17a6e89b19d85aa4e0e9"
server_key_version: "encryptionkey@0"
I20260620 08:07:15.651782 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "0858f2fd1499472d98b92560f321f549"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "ab360d9dac96ba40ad6954fe12f48fac"
server_key_iv: "51d31b22099f17a6e89b19d85aa4e0e9"
server_key_version: "encryptionkey@0"
I20260620 08:07:15.653350 15098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:07:15.654353 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:15.654681 15098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:15.654771 15098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "0858f2fd1499472d98b92560f321f549"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "ab360d9dac96ba40ad6954fe12f48fac"
server_key_iv: "51d31b22099f17a6e89b19d85aa4e0e9"
server_key_version: "encryptionkey@0"
I20260620 08:07:15.654872 15098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:15.674888 15098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:15.677531 15098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:15.677759 15098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:15.682157 15098 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:36017
I20260620 08:07:15.682193 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:36017 every 8 connection(s)
I20260620 08:07:15.682590 15098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:07:15.683612 15166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:15.686221 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap starting.
I20260620 08:07:15.686931 15166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:15.687276 15166 log.cc:824] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:15.688127 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: No bootstrap required, opened a new log
I20260620 08:07:15.689960 15166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:15.690110 15166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:15.690132 15166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0858f2fd1499472d98b92560f321f549, State: Initialized, Role: FOLLOWER
I20260620 08:07:15.690304 15166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:15.690384 15166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:07:15.690408 15166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:07:15.690438 15166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:15.691057 15166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:15.691148 15166 leader_election.cc:302] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0858f2fd1499472d98b92560f321f549; no voters:
I20260620 08:07:15.691355 15166 leader_election.cc:288] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:07:15.691480 15169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:07:15.691677 15166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:07:15.691762 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15098
I20260620 08:07:15.691720 15169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 LEADER]: Becoming Leader. State: Replica: 0858f2fd1499472d98b92560f321f549, State: Running, Role: LEADER
I20260620 08:07:15.691866 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:07:15.691888 15169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:15.692072 32207 external_mini_cluster.cc:1428] Setting key 811c27b786bc906a87437ed438dea586
I20260620 08:07:15.692312 15171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0858f2fd1499472d98b92560f321f549. Latest consensus state: current_term: 1 leader_uuid: "0858f2fd1499472d98b92560f321f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } } }
I20260620 08:07:15.692436 15171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: This master's current role is: LEADER
I20260620 08:07:15.692617 15170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0858f2fd1499472d98b92560f321f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } } }
I20260620 08:07:15.692701 15170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: This master's current role is: LEADER
I20260620 08:07:15.693255 15177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:07:15.693953 15177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:07:15.695641 15177 catalog_manager.cc:1347] Generated new cluster ID: e082f6d7a3a0455982658fea4d289397
I20260620 08:07:15.695698 15177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:07:15 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:15.703720 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:15.692773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.31.115.254:36017)
0620 08:07:15.692965 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:15.692976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:15.693252 (+ 276us) server_negotiation.cc:407] Connection header received
0620 08:07:15.693513 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:15.693523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:15.693589 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:15.693729 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:15.694525 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:15.695817 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:15.696575 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:15.697009 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:15.697124 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:15.699628 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:15.699632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:15.699633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:15.699652 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:15.701984 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:15.702466 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:15.702470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:15.702470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:15.702571 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:15.702788 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:15.702789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:15.702789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:15.703063 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:15.703219 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:15.703335 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 08:07:15.703487 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260620 08:07:15.709041 15177 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:07:15.709744 15177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:07:15.714866 15177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Generated new TSK 0
I20260620 08:07:15.715095 15177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:07:15.749595 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36017
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:15.855142 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:15.855361 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:15.855387 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:15.856734 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:15.856786 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:15.856813 15192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:07:15.858606 15192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:15.858871 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:15.859186 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:15.864629 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:15.864776 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:15.864629 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:15.864670 15192 server_base.cc:1034] running on GCE node
I20260620 08:07:15.865115 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:15.865402 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:15.866608 15192 hybrid_clock.cc:648] HybridClock initialized: now 1781942835866583 us; error 35 us; skew 500 ppm
Jun 20 08:07:15 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942835, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:15.869421 15192 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:07:15.870015 15192 webserver.cc:460] Webserver started at http://127.31.115.193:39475/ using document root <none> and password file <none>
I20260620 08:07:15.870273 15192 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:15.870342 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:15.870504 15192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:07:15.871996 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "643fa26c0e624a6599b683cabfba643d"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "4165884cb36b54941e38d7e4dc2ed399"
server_key_iv: "c1d98e7e8ef26a8866384d0b59a40a4e"
server_key_version: "encryptionkey@0"
I20260620 08:07:15.872592 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "643fa26c0e624a6599b683cabfba643d"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "4165884cb36b54941e38d7e4dc2ed399"
server_key_iv: "c1d98e7e8ef26a8866384d0b59a40a4e"
server_key_version: "encryptionkey@0"
I20260620 08:07:15.874339 15192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 08:07:15.875609 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:15.875908 15192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:15.876160 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "643fa26c0e624a6599b683cabfba643d"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "4165884cb36b54941e38d7e4dc2ed399"
server_key_iv: "c1d98e7e8ef26a8866384d0b59a40a4e"
server_key_version: "encryptionkey@0"
I20260620 08:07:15.876284 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:15.899983 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:15.902415 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:15.902617 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:15.902889 15192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:15.903255 15192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:15.903322 15192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:15.903364 15192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:15.903409 15192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:15.910081 15192 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:45363
I20260620 08:07:15.910131 15320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:45363 every 8 connection(s)
I20260620 08:07:15.910476 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:07:15.917265 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15192
I20260620 08:07:15.917428 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
Jun 20 08:07:15 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942835, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:15.917686 32207 external_mini_cluster.cc:1428] Setting key 6b4fa26699417ebe3412fdcef604f9b3
I20260620 08:07:15.921675 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:15.911295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:46215 (local address 127.31.115.254:36017)
0620 08:07:15.911376 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:15.911378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:15.911697 (+ 319us) server_negotiation.cc:407] Connection header received
0620 08:07:15.912205 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:15.912206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:15.912223 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:15.912295 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:15.913574 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:15.914118 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:15.915184 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:15.915413 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:15.915479 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:15.918195 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:15.918197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:15.918197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:15.918210 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:15.919907 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:15.920383 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:15.920384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:15.920385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:15.920458 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:15.920831 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:15.920832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:15.920832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:15.920994 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:15.921075 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:15.921471 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 08:07:15.921550 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 08:07:15.921952 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:15.911282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:46215)
0620 08:07:15.911528 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:15.911577 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:15.912080 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:15.912564 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:15.912568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:15.913209 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:15.913460 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:15.913465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:15.914220 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:15.914221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:15.915086 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:15.915090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:15.915658 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:15.915659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:15.915777 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:15.916344 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:15.916354 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:15.918084 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:15.920014 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:15.920024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:15.920026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:15.920289 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:15.920566 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:15.920566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:15.920567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:15.920743 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:15.921077 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:15.921084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:15.921237 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 08:07:15.921399 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 08:07:15.921559 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
I20260620 08:07:15.922518 15321 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36017
I20260620 08:07:15.922639 15321 heartbeater.cc:464] Registering TS with master...
I20260620 08:07:15.922869 15321 heartbeater.cc:511] Master 127.31.115.254:36017 requested a full tablet report, sending...
I20260620 08:07:15.923560 15130 ts_manager.cc:194] Registered new tserver with Master: 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363)
I20260620 08:07:15.924234 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:46215
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:07:15.950825 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36017
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:16.040489 15326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:16.040747 15326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:16.040798 15326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:16.042536 15326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:16.042626 15326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:16.042676 15326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:07:16.044360 15326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:16.044689 15326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:16.044946 15326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:16.050907 15331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:16.050918 15332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:16.051101 15326 server_base.cc:1034] running on GCE node
W20260620 08:07:16.050963 15334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:16.051702 15326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:16.051972 15326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:16.053164 15326 hybrid_clock.cc:648] HybridClock initialized: now 1781942836053149 us; error 26 us; skew 500 ppm
Jun 20 08:07:16 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942836, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:16.056627 15326 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:07:16.057124 15326 webserver.cc:460] Webserver started at http://127.31.115.194:45559/ using document root <none> and password file <none>
I20260620 08:07:16.057394 15326 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:16.057475 15326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:16.057682 15326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:07:16.058925 15326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "bcc72036ca16f9f227a09bf97b99a6eb"
server_key_iv: "f48583299173510390665e4fd0bd4e93"
server_key_version: "encryptionkey@0"
I20260620 08:07:16.059355 15326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "bcc72036ca16f9f227a09bf97b99a6eb"
server_key_iv: "f48583299173510390665e4fd0bd4e93"
server_key_version: "encryptionkey@0"
I20260620 08:07:16.060802 15326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:07:16.061834 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:16.062029 15326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:16.062122 15326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "bcc72036ca16f9f227a09bf97b99a6eb"
server_key_iv: "f48583299173510390665e4fd0bd4e93"
server_key_version: "encryptionkey@0"
I20260620 08:07:16.062188 15326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:16.077580 15326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:16.080262 15326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:16.080399 15326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:16.080659 15326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:16.081048 15326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:16.081099 15326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:16.081125 15326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:16.081139 15326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:16.088055 15326 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:37853
I20260620 08:07:16.088171 15454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:37853 every 8 connection(s)
I20260620 08:07:16.088596 15326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:07:16 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942836, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:16.096568 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15326
I20260620 08:07:16.096700 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:07:16.096961 32207 external_mini_cluster.cc:1428] Setting key 96ed0a1ce03cd3d80d8ab1d351b38cc1
I20260620 08:07:16.102206 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.089477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:44823)
0620 08:07:16.089823 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.089863 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.090395 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.090856 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.090861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.091639 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:16.091876 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.091882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.092855 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.092856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.093935 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.093939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.094396 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.094397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.094559 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.095248 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:16.095258 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:16.097320 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:16.100029 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:16.100045 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:16.100048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:16.100362 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:16.100642 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:16.100643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:16.100643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:16.100801 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:16.101215 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:16.101224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:16.101399 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.101841 (+ 442us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.102012 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":88,"threads_started":1}
I20260620 08:07:16.102283 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.089459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:44823 (local address 127.31.115.254:36017)
0620 08:07:16.089553 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.089556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.089971 (+ 415us) server_negotiation.cc:407] Connection header received
0620 08:07:16.090562 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.090564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.090578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.090635 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:16.092031 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.092744 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.094049 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.094281 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.094381 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.097533 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:16.097535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:16.097535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:16.097544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:16.099594 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:16.100483 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:16.100487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:16.100487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:16.100564 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:16.100885 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:16.100886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:16.100887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:16.101029 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:16.101142 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.101894 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.102135 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:07:16.103086 15455 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36017
I20260620 08:07:16.103238 15455 heartbeater.cc:464] Registering TS with master...
I20260620 08:07:16.103487 15455 heartbeater.cc:511] Master 127.31.115.254:36017 requested a full tablet report, sending...
I20260620 08:07:16.103960 15130 ts_manager.cc:194] Registered new tserver with Master: 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853)
I20260620 08:07:16.104493 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:44823
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:07:16.129145 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36017
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:16.221817 15460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:16.222059 15460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:16.222087 15460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:16.223393 15460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:16.223449 15460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:16.223479 15460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:07:16.225090 15460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:16.225324 15460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:16.225534 15460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:16.231045 15466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:16.231566 15465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:16.231693 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:16.232067 15460 server_base.cc:1034] running on GCE node
I20260620 08:07:16.232336 15460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:16.232650 15460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:16.233860 15460 hybrid_clock.cc:648] HybridClock initialized: now 1781942836233831 us; error 47 us; skew 500 ppm
Jun 20 08:07:16 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942836, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:16.237232 15460 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:07:16.237859 15460 webserver.cc:460] Webserver started at http://127.31.115.195:35115/ using document root <none> and password file <none>
I20260620 08:07:16.238078 15460 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:16.238127 15460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:16.238229 15460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:07:16.239327 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "091f6815ebf545af85d800f146073d97"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "703d2af8d36c181341f024852ea97b6f"
server_key_iv: "cca89616c5de1c1ce1fbfe0eb0c5ed33"
server_key_version: "encryptionkey@0"
I20260620 08:07:16.239670 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "091f6815ebf545af85d800f146073d97"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "703d2af8d36c181341f024852ea97b6f"
server_key_iv: "cca89616c5de1c1ce1fbfe0eb0c5ed33"
server_key_version: "encryptionkey@0"
I20260620 08:07:16.241357 15460 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:07:16.242429 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:16.242673 15460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:16.242775 15460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "091f6815ebf545af85d800f146073d97"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "703d2af8d36c181341f024852ea97b6f"
server_key_iv: "cca89616c5de1c1ce1fbfe0eb0c5ed33"
server_key_version: "encryptionkey@0"
I20260620 08:07:16.242843 15460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:16.255992 15460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:16.258597 15460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:16.258723 15460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:16.258996 15460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:16.259488 15460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:16.259554 15460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:16.259598 15460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:16.259626 15460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:16.266412 15460 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41175
I20260620 08:07:16.266557 15588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41175 every 8 connection(s)
I20260620 08:07:16.266965 15460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:07:16 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942836, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:16.275223 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15460
I20260620 08:07:16.275349 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:07:16.275570 32207 external_mini_cluster.cc:1428] Setting key 5a1700d2f94632396bda0eaf04835145
I20260620 08:07:16.277952 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.267958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:34647 (local address 127.31.115.254:36017)
0620 08:07:16.268035 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.268038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.268242 (+ 204us) server_negotiation.cc:407] Connection header received
0620 08:07:16.268725 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.268726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.268738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.268798 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:16.269928 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.270456 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.271683 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.271902 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.271964 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.274523 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:16.274525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:16.274525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:16.274533 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:16.276289 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:16.276796 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:16.276798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:16.276798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:16.276873 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:16.277158 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:16.277159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:16.277159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:16.277304 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:16.277387 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.277756 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.277883 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:07:16.278107 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.267895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:34647)
0620 08:07:16.268109 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.268151 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.268617 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.268879 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.268882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.269501 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:16.269829 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.269835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.270660 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.270662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.271579 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.271582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.272010 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.272012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.272149 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.272745 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:16.272755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:16.274408 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:16.276407 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:16.276417 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:16.276420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:16.276684 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:16.276953 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:16.276955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:16.276955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:16.277080 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:16.277389 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:16.277398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:16.277549 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.277704 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.277874 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
I20260620 08:07:16.279033 15589 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36017
I20260620 08:07:16.279139 15589 heartbeater.cc:464] Registering TS with master...
I20260620 08:07:16.279433 15589 heartbeater.cc:511] Master 127.31.115.254:36017 requested a full tablet report, sending...
I20260620 08:07:16.279939 15130 ts_manager.cc:194] Registered new tserver with Master: 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
I20260620 08:07:16.280450 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:34647
I20260620 08:07:16.288219 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:07:16.296751 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.289693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.31.115.254:36017)
0620 08:07:16.289745 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.289747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.289812 (+ 65us) server_negotiation.cc:407] Connection header received
0620 08:07:16.289870 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.289871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.289880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.289928 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:16.290507 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.291052 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.291965 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.292163 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.292252 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.293801 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:16.293802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:16.293802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:16.293810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:16.295355 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:16.295769 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:16.295770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:16.295771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:16.295841 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:16.296114 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:16.296115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:16.296116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:16.296282 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:16.296412 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.296575 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.296643 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:07:16.298601 15130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:07:16.300240 15130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:07:16.307461 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.303200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.31.115.194:37853)
0620 08:07:16.303397 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.303401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.303430 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:07:16.303464 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.303465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.303477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.303665 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:16.303929 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.304689 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.306087 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.306931 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.307054 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.307114 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.307303 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.307347 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260620 08:07:16.307475 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.302901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41175 (local address 127.0.0.1:46094)
0620 08:07:16.303480 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.303490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.303530 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.303940 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.303941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.303943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:16.303980 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.303981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.305309 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.305310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.306378 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.306380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.307215 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.307216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.307288 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.307291 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.307342 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.307375 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260620 08:07:16.307475 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.302901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:37853 (local address 127.0.0.1:53102)
0620 08:07:16.303242 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.303288 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.303360 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.303761 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.303764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.303766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:16.303843 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.303846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.304804 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.304804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.305989 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.305992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.307071 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.307073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.307178 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.307197 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.307278 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.307348 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":77,"threads_started":1}
I20260620 08:07:16.307909 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.303389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.31.115.195:41175)
0620 08:07:16.303604 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.303676 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.303687 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:07:16.303729 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.303731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.303747 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.303863 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:16.304060 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.305193 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.306491 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.307006 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.307117 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.307179 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.307732 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.307787 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260620 08:07:16.308533 15523 tablet_service.cc:1467] Processing CreateTablet for tablet 945d8c7f84b548239f28351c40f9fe6b (DEFAULT_TABLE table=test-table [id=42e1aa0bb03445e8bb281dd132262851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:07:16.308535 15389 tablet_service.cc:1467] Processing CreateTablet for tablet 945d8c7f84b548239f28351c40f9fe6b (DEFAULT_TABLE table=test-table [id=42e1aa0bb03445e8bb281dd132262851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:07:16.308931 15523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945d8c7f84b548239f28351c40f9fe6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:16.308931 15389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945d8c7f84b548239f28351c40f9fe6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:16.312987 15607 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Bootstrap starting.
I20260620 08:07:16.313822 15606 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Bootstrap starting.
I20260620 08:07:16.313874 15607 tablet_bootstrap.cc:654] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:16.313949 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.302936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45363 (local address 127.0.0.1:48830)
0620 08:07:16.303620 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.303628 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.303672 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.303943 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.303944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.303946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:16.303992 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.303994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.306480 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.306480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.307784 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.307788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.313587 (+ 5799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.313589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.313696 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.313703 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.313773 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.313841 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":102,"threads_started":1}
I20260620 08:07:16.313983 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.303031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.31.115.193:45363)
0620 08:07:16.303620 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.303625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.303725 (+ 100us) server_negotiation.cc:407] Connection header received
0620 08:07:16.303769 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.303772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.303787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.303857 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:16.304090 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.305183 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.312789 (+ 7606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.313467 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.313547 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.313589 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.313828 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.313887 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":100,"threads_started":1}
I20260620 08:07:16.314713 15607 log.cc:824] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:16.315304 15255 tablet_service.cc:1467] Processing CreateTablet for tablet 945d8c7f84b548239f28351c40f9fe6b (DEFAULT_TABLE table=test-table [id=42e1aa0bb03445e8bb281dd132262851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:07:16.315639 15255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945d8c7f84b548239f28351c40f9fe6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:16.315698 15607 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: No bootstrap required, opened a new log
I20260620 08:07:16.315773 15607 ts_tablet_manager.cc:1397] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:07:16.317216 15607 raft_consensus.cc:348] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.317385 15607 raft_consensus.cc:374] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:16.317433 15607 raft_consensus.cc:729] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9f9d4ca7684ff1b960d7f259bf2f8d, State: Initialized, Role: FOLLOWER
I20260620 08:07:16.317552 15607 consensus_queue.cc:260] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.317924 15607 ts_tablet_manager.cc:1428] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:07:16.317947 15455 heartbeater.cc:503] Master 127.31.115.254:36017 was elected leader, sending a full tablet report...
I20260620 08:07:16.318441 15610 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Bootstrap starting.
I20260620 08:07:16.318645 15606 tablet_bootstrap.cc:654] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:16.319245 15610 tablet_bootstrap.cc:654] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:16.319271 15606 log.cc:824] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:16.319830 15610 log.cc:824] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:16.320111 15606 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: No bootstrap required, opened a new log
I20260620 08:07:16.320178 15606 ts_tablet_manager.cc:1397] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 08:07:16.320531 15610 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: No bootstrap required, opened a new log
I20260620 08:07:16.320616 15610 ts_tablet_manager.cc:1397] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:07:16.321817 15606 raft_consensus.cc:348] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.321969 15606 raft_consensus.cc:374] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:16.321992 15606 raft_consensus.cc:729] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091f6815ebf545af85d800f146073d97, State: Initialized, Role: FOLLOWER
I20260620 08:07:16.322118 15606 consensus_queue.cc:260] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.322302 15610 raft_consensus.cc:348] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.322414 15606 ts_tablet_manager.cc:1428] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:07:16.322449 15589 heartbeater.cc:503] Master 127.31.115.254:36017 was elected leader, sending a full tablet report...
I20260620 08:07:16.322515 15610 raft_consensus.cc:374] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:16.322569 15610 raft_consensus.cc:729] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643fa26c0e624a6599b683cabfba643d, State: Initialized, Role: FOLLOWER
I20260620 08:07:16.322678 15610 consensus_queue.cc:260] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.322991 15610 ts_tablet_manager.cc:1428] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:07:16.323033 15321 heartbeater.cc:503] Master 127.31.115.254:36017 was elected leader, sending a full tablet report...
W20260620 08:07:16.339402 15456 tablet.cc:2369] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:07:16.411389 15322 tablet.cc:2369] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:07:16.517853 15590 tablet.cc:2369] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:07:16.583995 15614 raft_consensus.cc:482] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:07:16.584151 15614 raft_consensus.cc:504] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.584592 15614 leader_election.cc:288] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
I20260620 08:07:16.589352 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.585019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:46531 (local address 127.31.115.195:41175)
0620 08:07:16.585063 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.585065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.585208 (+ 143us) server_negotiation.cc:407] Connection header received
0620 08:07:16.585250 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.585251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.585263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.585346 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:16.585687 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.586411 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.588174 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.588820 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.588883 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.588892 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.589175 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.589271 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:07:16.589344 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.584895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:37853 (local address 127.31.115.193:59093)
0620 08:07:16.585057 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.585070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.585146 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.585392 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.585393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.585398 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:16.585514 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.585517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.586542 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.586543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.588002 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.588007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.588928 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.588929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.589085 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.589094 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.589157 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.589214 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":185,"thread_start_us":86,"threads_started":1}
I20260620 08:07:16.589406 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.585000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:59093 (local address 127.31.115.194:37853)
0620 08:07:16.585054 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.585057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.585212 (+ 155us) server_negotiation.cc:407] Connection header received
0620 08:07:16.585250 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.585251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.585266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.585346 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:16.585646 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.586434 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.588174 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.588778 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.588867 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.588879 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.589274 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.589325 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:07:16.589830 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.584896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41175 (local address 127.31.115.193:46531)
0620 08:07:16.585056 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.585070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:16.585146 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:16.585392 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:16.585393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:16.585398 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:16.585520 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.585522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.586542 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.586543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.588002 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:16.588007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.588926 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.588928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:16.589085 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.589093 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 08:07:16.589169 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:07:16.589214 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":85,"thread_start_us":85,"threads_started":1}
I20260620 08:07:16.589986 15409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945d8c7f84b548239f28351c40f9fe6b" candidate_uuid: "643fa26c0e624a6599b683cabfba643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" is_pre_election: true
I20260620 08:07:16.590211 15409 raft_consensus.cc:2454] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 643fa26c0e624a6599b683cabfba643d in term 0.
I20260620 08:07:16.590492 15543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945d8c7f84b548239f28351c40f9fe6b" candidate_uuid: "643fa26c0e624a6599b683cabfba643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091f6815ebf545af85d800f146073d97" is_pre_election: true
I20260620 08:07:16.590591 15209 leader_election.cc:302] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f9f9d4ca7684ff1b960d7f259bf2f8d, 643fa26c0e624a6599b683cabfba643d; no voters:
I20260620 08:07:16.590668 15543 raft_consensus.cc:2454] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 643fa26c0e624a6599b683cabfba643d in term 0.
I20260620 08:07:16.590843 15614 raft_consensus.cc:2781] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:07:16.590924 15614 raft_consensus.cc:482] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:07:16.590950 15614 raft_consensus.cc:3037] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:16.592221 15614 raft_consensus.cc:504] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.592540 15614 leader_election.cc:288] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 1 election: Requested vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
I20260620 08:07:16.592592 15543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945d8c7f84b548239f28351c40f9fe6b" candidate_uuid: "643fa26c0e624a6599b683cabfba643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091f6815ebf545af85d800f146073d97"
I20260620 08:07:16.592653 15409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945d8c7f84b548239f28351c40f9fe6b" candidate_uuid: "643fa26c0e624a6599b683cabfba643d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d"
I20260620 08:07:16.592694 15543 raft_consensus.cc:3037] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:16.592757 15409 raft_consensus.cc:3037] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:16.593698 15543 raft_consensus.cc:2454] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 643fa26c0e624a6599b683cabfba643d in term 1.
I20260620 08:07:16.593724 15409 raft_consensus.cc:2454] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 643fa26c0e624a6599b683cabfba643d in term 1.
I20260620 08:07:16.593912 15208 leader_election.cc:302] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091f6815ebf545af85d800f146073d97, 643fa26c0e624a6599b683cabfba643d; no voters:
I20260620 08:07:16.594029 15614 raft_consensus.cc:2781] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:07:16.594206 15614 raft_consensus.cc:686] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 LEADER]: Becoming Leader. State: Replica: 643fa26c0e624a6599b683cabfba643d, State: Running, Role: LEADER
I20260620 08:07:16.594341 15614 consensus_queue.cc:237] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:16.595383 15130 catalog_manager.cc:5495] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d reported cstate change: term changed from 0 to 1, leader changed from <none> to 643fa26c0e624a6599b683cabfba643d (127.31.115.193). New cstate: current_term: 1 leader_uuid: "643fa26c0e624a6599b683cabfba643d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } health_report { overall_health: HEALTHY } } }
I20260620 08:07:16.664966 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.661755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.31.115.193:45363)
0620 08:07:16.661815 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:16.661818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:16.661846 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:07:16.661916 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:16.661917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:16.661938 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:16.662031 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:07:16.662204 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.662763 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.663925 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:16.664123 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:16.664199 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:16.664468 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:07:16.664688 (+ 220us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:07:16.664779 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:16.664870 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 08:07:16.664893 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:07:16.667181 15409 raft_consensus.cc:1261] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Refusing update from remote peer 643fa26c0e624a6599b683cabfba643d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:07:16.667217 15543 raft_consensus.cc:1261] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Refusing update from remote peer 643fa26c0e624a6599b683cabfba643d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:07:16.667654 15617 consensus_queue.cc:1035] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:07:16.667804 15614 consensus_queue.cc:1035] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [LEADER]: Connected to new peer: Peer: permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:07:16.671344 15620 mvcc.cc:205] Tried to move back new op lower bound from 7298837858985631744 to 7298837858691092480. Current Snapshot: MvccSnapshot[applied={T|T < 7298837858985631744 or (T in {7298837858985631744})}]
I20260620 08:07:16.673826 15130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:07:16.673957 15130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:07:16.675220 15130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Sending DeleteTablet for 3 replicas of tablet 945d8c7f84b548239f28351c40f9fe6b
I20260620 08:07:16.675695 15255 tablet_service.cc:1514] Processing DeleteTablet for tablet 945d8c7f84b548239f28351c40f9fe6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:48830
I20260620 08:07:16.675724 15523 tablet_service.cc:1514] Processing DeleteTablet for tablet 945d8c7f84b548239f28351c40f9fe6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46094
I20260620 08:07:16.675997 15389 tablet_service.cc:1514] Processing DeleteTablet for tablet 945d8c7f84b548239f28351c40f9fe6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53102
I20260620 08:07:16.676051 15631 tablet_replica.cc:330] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: stopping tablet replica
I20260620 08:07:16.676177 15631 raft_consensus.cc:2229] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:07:16.676263 15632 tablet_replica.cc:330] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: stopping tablet replica
I20260620 08:07:16.676384 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15192
I20260620 08:07:16.676406 15632 raft_consensus.cc:2229] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:07:16.676525 15632 raft_consensus.cc:2258] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:07:16.676440 15631 raft_consensus.cc:2258] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:07:16.678222 15631 ts_tablet_manager.cc:1905] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:07:16.678470 15632 ts_tablet_manager.cc:1905] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:07:16.680645 15631 ts_tablet_manager.cc:1918] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:07:16.680742 15631 log.cc:1196] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/945d8c7f84b548239f28351c40f9fe6b
I20260620 08:07:16.680784 15632 ts_tablet_manager.cc:1918] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:07:16.680898 15632 log.cc:1196] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/945d8c7f84b548239f28351c40f9fe6b
I20260620 08:07:16.681056 15631 ts_tablet_manager.cc:1939] T 945d8c7f84b548239f28351c40f9fe6b P 091f6815ebf545af85d800f146073d97: Deleting consensus metadata
I20260620 08:07:16.681250 15632 ts_tablet_manager.cc:1939] T 945d8c7f84b548239f28351c40f9fe6b P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Deleting consensus metadata
I20260620 08:07:16.681803 15114 catalog_manager.cc:4841] TS 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): tablet 945d8c7f84b548239f28351c40f9fe6b (table test-table [id=42e1aa0bb03445e8bb281dd132262851]) successfully deleted
I20260620 08:07:16.681914 15115 catalog_manager.cc:4841] TS 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): tablet 945d8c7f84b548239f28351c40f9fe6b (table test-table [id=42e1aa0bb03445e8bb281dd132262851]) successfully deleted
W20260620 08:07:16.685757 15117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:45363 (error 108)
I20260620 08:07:16.686096 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15326
I20260620 08:07:16.686815 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:16.686319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45363 (local address 127.0.0.1:48846)
0620 08:07:16.686709 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:16.686764 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
W20260620 08:07:16.687037 15117 catalog_manager.cc:4584] TS 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 945d8c7f84b548239f28351c40f9fe6b: Network error: Client connection negotiation failed: client connection to 127.31.115.193:45363: connect: Connection refused (error 111)
I20260620 08:07:16.692729 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15460
I20260620 08:07:16.698520 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15098
I20260620 08:07:36.705648 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36017
--webserver_interface=127.31.115.254
--webserver_port=38925
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:36.799022 15634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:36.799263 15634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:36.799291 15634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:36.801324 15634 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:07:36.801419 15634 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:07:36.801450 15634 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:07:36.801476 15634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:07:36.801505 15634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:36.801532 15634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:36.801559 15634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:07:36.801586 15634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:07:36.804594 15634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:36.804917 15634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:36.805225 15634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:07:36.816236 15634 server_base.cc:1034] running on GCE node
W20260620 08:07:36.816229 15641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:36.816233 15640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:36.816497 15643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:36.816856 15634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:36.817272 15634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:36.818468 15634 hybrid_clock.cc:648] HybridClock initialized: now 1781942856818441 us; error 68 us; skew 500 ppm
Jun 20 08:07:36 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942856, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:36.824656 15634 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:07:36.825230 15634 webserver.cc:460] Webserver started at http://127.31.115.254:38925/ using document root <none> and password file <none>
I20260620 08:07:36.825510 15634 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:36.825659 15634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:36.827286 15634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:36.828186 15650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:36.828527 15634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:36.828648 15634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "0858f2fd1499472d98b92560f321f549"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "ab360d9dac96ba40ad6954fe12f48fac"
server_key_iv: "51d31b22099f17a6e89b19d85aa4e0e9"
server_key_version: "encryptionkey@0"
I20260620 08:07:36.829093 15634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:36.857506 15634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:36.860148 15634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:36.860430 15634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:36.865345 15634 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:36017
I20260620 08:07:36.865420 15702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:36017 every 8 connection(s)
I20260620 08:07:36.865885 15634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:07:36.867568 15703 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:07:36.868218 15703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap starting.
I20260620 08:07:36.872985 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15634
I20260620 08:07:36.873395 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:45363
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:36017
--webserver_port=39475
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:07:36.878595 15703 log.cc:824] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:36.881978 15703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:07:36.882421 15703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap complete.
I20260620 08:07:36.886776 15703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:36.887126 15703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0858f2fd1499472d98b92560f321f549, State: Initialized, Role: FOLLOWER
I20260620 08:07:36.887288 15703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:36.887360 15703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:07:36.887403 15703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:07:36.887445 15703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:07:36.888957 15703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:36.889087 15703 leader_election.cc:302] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0858f2fd1499472d98b92560f321f549; no voters:
I20260620 08:07:36.889274 15703 leader_election.cc:288] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:07:36.889432 15707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:07:36.889533 15703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:07:36.889681 15707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 LEADER]: Becoming Leader. State: Replica: 0858f2fd1499472d98b92560f321f549, State: Running, Role: LEADER
I20260620 08:07:36.889811 15707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:36.890547 15707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0858f2fd1499472d98b92560f321f549. Latest consensus state: current_term: 2 leader_uuid: "0858f2fd1499472d98b92560f321f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } } }
I20260620 08:07:36.890724 15707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: This master's current role is: LEADER
I20260620 08:07:36.890766 15708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0858f2fd1499472d98b92560f321f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } } }
I20260620 08:07:36.890828 15708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: This master's current role is: LEADER
I20260620 08:07:36.891673 15721 catalog_manager.cc:1259] Loaded cluster ID: e082f6d7a3a0455982658fea4d289397
I20260620 08:07:36.891722 15721 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: loading cluster ID for follower catalog manager: success
I20260620 08:07:36.893018 15721 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: acquiring CA information for follower catalog manager: success
I20260620 08:07:36.893579 15721 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:07:36.893702 15722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:07:36.894155 15722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:07:36.894275 15722 catalog_manager.cc:1259] Loaded cluster ID: e082f6d7a3a0455982658fea4d289397
I20260620 08:07:36.894315 15722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:07:36.894544 15722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:07:36.894640 15722 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Loaded TSK: 0
I20260620 08:07:36.911572 15722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Generated new TSK 1
I20260620 08:07:36.911723 15722 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:07:36.974655 15705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:36.974886 15705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:36.974912 15705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:36.976298 15705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:36.976357 15705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:36.976387 15705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:07:36.978135 15705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:45363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=39475
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:36.978425 15705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:36.978708 15705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:36.983953 15733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:36.983975 15731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:36.984014 15730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:36.984601 15705 server_base.cc:1034] running on GCE node
I20260620 08:07:36.984827 15705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:36.985078 15705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:36.986284 15705 hybrid_clock.cc:648] HybridClock initialized: now 1781942856986266 us; error 32 us; skew 500 ppm
Jun 20 08:07:36 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942856, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:36.989347 15705 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:07:36.989982 15705 webserver.cc:460] Webserver started at http://127.31.115.193:39475/ using document root <none> and password file <none>
I20260620 08:07:36.990240 15705 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:36.990299 15705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:36.991899 15705 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:07:36.992674 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:36.992887 15705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:36.992980 15705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "643fa26c0e624a6599b683cabfba643d"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "4165884cb36b54941e38d7e4dc2ed399"
server_key_iv: "c1d98e7e8ef26a8866384d0b59a40a4e"
server_key_version: "encryptionkey@0"
I20260620 08:07:36.993355 15705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:37.002784 15705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:37.005125 15705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:37.005295 15705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:37.005599 15705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:37.006156 15747 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:07:37.007704 15705 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:07:37.007762 15705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:07:37.007791 15705 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:07:37.008515 15705 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:07:37.008562 15705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:07:37.008610 15747 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Bootstrap starting.
I20260620 08:07:37.017565 15705 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:45363
I20260620 08:07:37.018216 15705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:07:37.019143 15854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:45363 every 8 connection(s)
I20260620 08:07:37.020677 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15705
I20260620 08:07:37.021004 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:37853
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:36017
--webserver_port=45559
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:07:37.022153 15747 log.cc:824] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:37.027331 15747 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:07:37.027776 15747 tablet_bootstrap.cc:492] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Bootstrap complete.
Jun 20 08:07:37 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942856, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:37.029214 15747 ts_tablet_manager.cc:1397] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20260620 08:07:37.031144 15747 raft_consensus.cc:348] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.031590 15747 raft_consensus.cc:729] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643fa26c0e624a6599b683cabfba643d, State: Initialized, Role: FOLLOWER
I20260620 08:07:37.031795 15747 consensus_queue.cc:260] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.032239 15747 ts_tablet_manager.cc:1428] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:07:37.033300 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.018955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:35243)
0620 08:07:37.019470 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.019528 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.020314 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.021229 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.021234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.021711 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:37.022019 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.022027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.023413 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.023415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.024790 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.024795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.025769 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.025770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.025909 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.026749 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:37.026760 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:37.028714 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:37.031213 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.031220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.031222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.031527 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.031822 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.031823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.031823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.032035 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.032591 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:37.032598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:37.032750 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.032866 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.033110 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":107,"threads_started":1}
I20260620 08:07:37.033432 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.019060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:35243 (local address 127.31.115.254:36017)
0620 08:07:37.020592 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.020604 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.020649 (+ 45us) server_negotiation.cc:407] Connection header received
0620 08:07:37.020801 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.020809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.020888 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.021046 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:37.022173 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.023251 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.024914 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.025487 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.025574 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.028863 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:37.028869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:37.028869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:37.028898 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:37.031094 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.031625 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.031629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.031629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.031749 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.032121 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.032122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.032123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.032471 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:37.032533 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.032996 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.033195 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1457,"thread_start_us":98,"threads_started":1}
I20260620 08:07:37.034132 15855 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36017
I20260620 08:07:37.034243 15855 heartbeater.cc:464] Registering TS with master...
I20260620 08:07:37.034485 15855 heartbeater.cc:511] Master 127.31.115.254:36017 requested a full tablet report, sending...
I20260620 08:07:37.035391 15667 ts_manager.cc:194] Registered new tserver with Master: 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363)
I20260620 08:07:37.036618 15667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:35243
I20260620 08:07:37.037757 15855 heartbeater.cc:503] Master 127.31.115.254:36017 was elected leader, sending a full tablet report...
Jun 20 08:07:37 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942856, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.193@KRBTEST.COM
I20260620 08:07:37.045112 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.036302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.31.115.193:45363)
0620 08:07:37.037020 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.037025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.037035 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:07:37.037067 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.037069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.037084 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.037205 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:37.037671 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.038590 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.039364 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.039672 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.039765 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.041973 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:37.041981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:37.041982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:37.041988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:37.043605 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.044109 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.044112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.044112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.044188 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.044463 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.044464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.044464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.044633 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:37.044711 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.044940 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.045038 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":72,"threads_started":1}
I20260620 08:07:37.045120 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.036050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45363 (local address 127.0.0.1:46736)
0620 08:07:37.036749 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.036785 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.036852 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.037213 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.037219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.037504 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:37.037552 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.037554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.038685 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.038686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.039267 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.039269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.039768 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.039769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.039850 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.040401 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:37.040406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:37.041868 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:37.043699 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.043709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.043711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.044001 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.044267 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.044268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.044268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.044335 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.044707 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:37.044715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:37.044774 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.044871 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.045008 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":91,"threads_started":1}
I20260620 08:07:37.045661 15789 tablet_service.cc:1514] Processing DeleteTablet for tablet 945d8c7f84b548239f28351c40f9fe6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:16 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46736
I20260620 08:07:37.046327 15864 tablet_replica.cc:330] stopping tablet replica
I20260620 08:07:37.046482 15864 raft_consensus.cc:2229] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:07:37.046586 15864 raft_consensus.cc:2258] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:07:37.046901 15864 ts_tablet_manager.cc:1905] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:07:37.049315 15864 ts_tablet_manager.cc:1918] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:07:37.049422 15864 log.cc:1196] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal/wals/945d8c7f84b548239f28351c40f9fe6b
I20260620 08:07:37.049738 15864 ts_tablet_manager.cc:1939] T 945d8c7f84b548239f28351c40f9fe6b P 643fa26c0e624a6599b683cabfba643d: Deleting consensus metadata
I20260620 08:07:37.050357 15654 catalog_manager.cc:4841] TS 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363): tablet 945d8c7f84b548239f28351c40f9fe6b (table test-table [id=42e1aa0bb03445e8bb281dd132262851]) successfully deleted
W20260620 08:07:37.119406 15859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:37.119644 15859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:37.119675 15859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:37.120939 15859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:37.120986 15859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:37.121017 15859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:07:37.122547 15859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:37853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=45559
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:37.122781 15859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:37.122980 15859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:37.128484 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:37.128479 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:37.128476 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:37.128810 15859 server_base.cc:1034] running on GCE node
I20260620 08:07:37.129061 15859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:37.129340 15859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:37.130534 15859 hybrid_clock.cc:648] HybridClock initialized: now 1781942857130511 us; error 37 us; skew 500 ppm
Jun 20 08:07:37 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942857, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:37.133577 15859 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:07:37.134203 15859 webserver.cc:460] Webserver started at http://127.31.115.194:45559/ using document root <none> and password file <none>
I20260620 08:07:37.134475 15859 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:37.134559 15859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:37.136071 15859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:07:37.136862 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:37.137100 15859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:37.137202 15859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "bcc72036ca16f9f227a09bf97b99a6eb"
server_key_iv: "f48583299173510390665e4fd0bd4e93"
server_key_version: "encryptionkey@0"
I20260620 08:07:37.137602 15859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:37.147286 15859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:37.149816 15859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:37.150003 15859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:37.150319 15859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:37.150728 15859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:37.150772 15859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:37.150810 15859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:37.150830 15859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:37.156862 15859 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:37853
I20260620 08:07:37.156913 15992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:37853 every 8 connection(s)
I20260620 08:07:37.157258 15859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:07:37.158100 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15859
I20260620 08:07:37.158437 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:41175
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:36017
--webserver_port=35115
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:07:37 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942857, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:37.169198 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.158091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:44203 (local address 127.31.115.254:36017)
0620 08:07:37.158179 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.158181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.158380 (+ 199us) server_negotiation.cc:407] Connection header received
0620 08:07:37.159038 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.159039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.159054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.159133 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:37.160505 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.161007 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.162652 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.162895 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.162964 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.165855 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:37.165856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:37.165857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:37.165866 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:37.167565 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.168065 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.168067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.168068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.168132 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.168436 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.168437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.168438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.168569 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:37.168656 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.168983 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.169091 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:07:37.169397 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.158065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:44203)
0620 08:07:37.158260 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.158303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.158927 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.159186 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.159190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.159894 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:37.160195 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.160202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.161125 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.161127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.162544 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.162549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.163025 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.163026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.163198 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.163801 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:37.163814 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:37.165740 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:37.167686 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.167695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.167698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.167960 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.168216 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.168217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.168218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.168347 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.168647 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:37.168655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:37.168817 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.168948 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.169131 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260620 08:07:37.169981 15993 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36017
I20260620 08:07:37.170094 15993 heartbeater.cc:464] Registering TS with master...
I20260620 08:07:37.170282 15993 heartbeater.cc:511] Master 127.31.115.254:36017 requested a full tablet report, sending...
I20260620 08:07:37.170673 15667 ts_manager.cc:194] Registered new tserver with Master: 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853)
I20260620 08:07:37.171123 15667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:44203
W20260620 08:07:37.250147 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:37.250370 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:37.250401 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:37.251595 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:37.251636 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:37.251664 15996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:07:37.253135 15996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:41175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=35115
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:37.253341 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:37.253539 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:37.259231 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:37.259321 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:37.259279 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:37.259589 15996 server_base.cc:1034] running on GCE node
I20260620 08:07:37.259842 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:37.260140 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:37.261394 15996 hybrid_clock.cc:648] HybridClock initialized: now 1781942857261376 us; error 48 us; skew 500 ppm
Jun 20 08:07:37 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942857, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:37.264609 15996 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:07:37.265220 15996 webserver.cc:460] Webserver started at http://127.31.115.195:35115/ using document root <none> and password file <none>
I20260620 08:07:37.265512 15996 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:37.265610 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:37.267221 15996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:37.268072 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:37.268302 15996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:37.268411 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "091f6815ebf545af85d800f146073d97"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "703d2af8d36c181341f024852ea97b6f"
server_key_iv: "cca89616c5de1c1ce1fbfe0eb0c5ed33"
server_key_version: "encryptionkey@0"
I20260620 08:07:37.268816 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:37.287712 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:37.290778 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:37.290997 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:37.291235 15996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:37.291579 15996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:37.291615 15996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:37.291638 15996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:37.291651 15996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:37.297869 15996 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41175
I20260620 08:07:37.297950 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41175 every 8 connection(s)
I20260620 08:07:37.298311 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:07:37.304733 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 15996
Jun 20 08:07:37 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942857, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:07:37.310570 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.299123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:42889 (local address 127.31.115.254:36017)
0620 08:07:37.299210 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.299212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.299486 (+ 274us) server_negotiation.cc:407] Connection header received
0620 08:07:37.300023 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.300024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.300038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.300094 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:37.301103 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.301669 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.303098 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.303340 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.303445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.306235 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:37.306236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:37.306237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:37.306245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:37.308538 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.309205 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.309207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.309207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.309280 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.309688 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.309689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.309689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.309801 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:37.309860 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.310293 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.310405 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:07:37.310681 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.299136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:42889)
0620 08:07:37.299311 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.299349 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.299878 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.300170 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.300174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.300783 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:37.300995 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.301000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.301777 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.301777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.302942 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.302946 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.303457 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.303458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.303676 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.304294 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:37.304303 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:37.306130 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:37.308646 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.308657 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.308660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.309102 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.309382 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:37.309384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:37.309384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:37.309570 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:37.309875 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:37.309883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:37.310098 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.310240 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.310436 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260620 08:07:37.311275 16125 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36017
I20260620 08:07:37.311399 16125 heartbeater.cc:464] Registering TS with master...
I20260620 08:07:37.311638 16125 heartbeater.cc:511] Master 127.31.115.254:36017 requested a full tablet report, sending...
I20260620 08:07:37.312176 15667 ts_manager.cc:194] Registered new tserver with Master: 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
I20260620 08:07:37.312681 15667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:42889
I20260620 08:07:37.314832 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.305246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.31.115.254:36017)
0620 08:07:37.305476 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.305478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.305600 (+ 122us) server_negotiation.cc:407] Connection header received
0620 08:07:37.305708 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.305709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.305722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.305853 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:37.306797 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.307445 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.308442 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.308746 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.308808 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.310497 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:37.310498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:37.310499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:37.310506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:37.312504 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.313343 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.313346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.313346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.313456 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.313844 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.313845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.313845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.314091 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:37.314517 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.314583 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.314720 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
I20260620 08:07:37.315999 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:07:37.318868 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.316372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.31.115.254:36017)
0620 08:07:37.316414 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.316416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.316507 (+ 91us) server_negotiation.cc:407] Connection header received
0620 08:07:37.316573 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.316574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.316604 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.316680 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:07:37.316771 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.317292 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.318103 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.318316 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.318415 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.318560 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:07:37.318607 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 08:07:37.318776 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48346: token expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:07:37.318948 16128 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48346: token expired
W20260620 08:07:37.319027 15592 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.115.254:36017)
I20260620 08:07:37.326023 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.319345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.31.115.254:36017)
0620 08:07:37.319399 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.319401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.319409 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:07:37.319437 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.319437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.319445 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.319540 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:37.320390 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.320885 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.321727 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.321926 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.321988 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.323324 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:37.323326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:37.323326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:37.323333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:37.324765 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.325174 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.325175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.325175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.325233 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:37.325475 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:37.325476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:37.325476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:37.325593 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:37.325747 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.325888 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.325952 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:07:37.327440 15667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:07:37.329082 15667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:07:37.331456 15789 tablet_service.cc:1467] Processing CreateTablet for tablet e693fd10f8144fb29c2e0c9ceff262c2 (DEFAULT_TABLE table=test-table [id=91ddf7884c9c41079179f75a74277ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:07:37.331691 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e693fd10f8144fb29c2e0c9ceff262c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:37.333233 15747 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Bootstrap starting.
I20260620 08:07:37.333627 15747 tablet_bootstrap.cc:654] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:37.335047 15747 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: No bootstrap required, opened a new log
I20260620 08:07:37.335116 15747 ts_tablet_manager.cc:1397] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:07:37.335572 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.331223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41175 (local address 127.0.0.1:54146)
0620 08:07:37.331635 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.331648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.331712 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.332066 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.332067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.332069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:37.332097 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.332097 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.333365 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.333366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.334354 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.334357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.335284 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.335285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.335348 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.335373 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.335442 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.335493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260620 08:07:37.335523 15747 raft_consensus.cc:348] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.335602 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.331453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.31.115.195:41175)
0620 08:07:37.331647 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.331650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.331706 (+ 56us) server_negotiation.cc:407] Connection header received
0620 08:07:37.331783 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.331785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.331800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.331907 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:37.332182 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.333262 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.334475 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.335207 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.335286 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.335359 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.335481 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.335524 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260620 08:07:37.335624 15747 raft_consensus.cc:374] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:37.335690 15747 raft_consensus.cc:729] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643fa26c0e624a6599b683cabfba643d, State: Initialized, Role: FOLLOWER
I20260620 08:07:37.335759 15747 consensus_queue.cc:260] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.335901 15747 ts_tablet_manager.cc:1428] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:37.336086 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.331286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.31.115.194:37853)
0620 08:07:37.331484 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.331489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.331500 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:07:37.331700 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.331703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.331725 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.331903 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:37.332046 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.333231 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.334568 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.335384 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.335846 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.335917 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.335949 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.335998 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260620 08:07:37.335764 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.331150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:37853 (local address 127.0.0.1:59588)
0620 08:07:37.331213 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.331226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.331271 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.331917 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.331918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.331920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:37.331977 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.331979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.333519 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.333520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.334474 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.334477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.335493 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.335495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.335573 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.335579 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.335615 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.335683 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260620 08:07:37.336892 15927 tablet_service.cc:1467] Processing CreateTablet for tablet e693fd10f8144fb29c2e0c9ceff262c2 (DEFAULT_TABLE table=test-table [id=91ddf7884c9c41079179f75a74277ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:07:37.336923 16059 tablet_service.cc:1467] Processing CreateTablet for tablet e693fd10f8144fb29c2e0c9ceff262c2 (DEFAULT_TABLE table=test-table [id=91ddf7884c9c41079179f75a74277ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:07:37.337249 15927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e693fd10f8144fb29c2e0c9ceff262c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:37.337265 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e693fd10f8144fb29c2e0c9ceff262c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:07:37.340379 16135 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Bootstrap starting.
I20260620 08:07:37.340533 16136 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Bootstrap starting.
I20260620 08:07:37.341138 16135 tablet_bootstrap.cc:654] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:37.341154 16136 tablet_bootstrap.cc:654] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Neither blocks nor log segments found. Creating new log.
I20260620 08:07:37.341748 16135 log.cc:824] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:37.341751 16136 log.cc:824] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:37.342790 16135 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: No bootstrap required, opened a new log
I20260620 08:07:37.342801 16136 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: No bootstrap required, opened a new log
I20260620 08:07:37.342876 16135 ts_tablet_manager.cc:1397] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:07:37.342876 16136 ts_tablet_manager.cc:1397] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:07:37.344937 16136 raft_consensus.cc:348] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.344937 16135 raft_consensus.cc:348] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.345146 16135 raft_consensus.cc:374] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:37.345146 16136 raft_consensus.cc:374] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:07:37.345186 16135 raft_consensus.cc:729] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9f9d4ca7684ff1b960d7f259bf2f8d, State: Initialized, Role: FOLLOWER
I20260620 08:07:37.345186 16136 raft_consensus.cc:729] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091f6815ebf545af85d800f146073d97, State: Initialized, Role: FOLLOWER
I20260620 08:07:37.345315 16136 consensus_queue.cc:260] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.345343 16135 consensus_queue.cc:260] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.345677 16136 ts_tablet_manager.cc:1428] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:07:37.345760 16135 ts_tablet_manager.cc:1428] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:07:37.345872 16125 heartbeater.cc:503] Master 127.31.115.254:36017 was elected leader, sending a full tablet report...
I20260620 08:07:37.345952 15993 heartbeater.cc:503] Master 127.31.115.254:36017 was elected leader, sending a full tablet report...
I20260620 08:07:37.384783 16140 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:07:37.384924 16140 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.385275 16140 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175), 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363)
I20260620 08:07:37.388679 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.385505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45363 (local address 127.31.115.194:50311)
0620 08:07:37.385627 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.385641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.385708 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.385880 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.385881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.385883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:37.385951 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.385953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.386733 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.386733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.387657 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.387660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.388309 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.388310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.388392 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.388396 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.388495 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.388543 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":49}
I20260620 08:07:37.388692 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.385642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:41217 (local address 127.31.115.195:41175)
0620 08:07:37.385705 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.385707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.386030 (+ 323us) server_negotiation.cc:407] Connection header received
0620 08:07:37.386102 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.386103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.386111 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.386149 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:37.386304 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.386882 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.387850 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.388221 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.388311 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.388321 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.388554 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.388631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:07:37.388921 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.385637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:50311 (local address 127.31.115.193:45363)
0620 08:07:37.385704 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.385707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.385719 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:07:37.385762 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.385763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.385792 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.385870 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:07:37.386026 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.386647 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.387752 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.388219 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.388312 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.388392 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.388788 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.388844 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:07:37.388684 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.385507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41175 (local address 127.31.115.194:41217)
0620 08:07:37.385947 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.385956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:37.386040 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:37.386169 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:37.386170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:37.386171 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:07:37.386241 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.386242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.386962 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.386963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.387761 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:37.387763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.388312 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.388313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:37.388384 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.388421 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:07:37.388507 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:07:37.388587 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":217,"thread_start_us":123,"threads_started":1}
I20260620 08:07:37.389184 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e693fd10f8144fb29c2e0c9ceff262c2" candidate_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643fa26c0e624a6599b683cabfba643d" is_pre_election: true
I20260620 08:07:37.389416 15809 raft_consensus.cc:2454] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f9f9d4ca7684ff1b960d7f259bf2f8d in term 0.
I20260620 08:07:37.389389 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e693fd10f8144fb29c2e0c9ceff262c2" candidate_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091f6815ebf545af85d800f146073d97" is_pre_election: true
I20260620 08:07:37.389541 16079 raft_consensus.cc:2454] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f9f9d4ca7684ff1b960d7f259bf2f8d in term 0.
I20260620 08:07:37.389773 15883 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f9f9d4ca7684ff1b960d7f259bf2f8d, 643fa26c0e624a6599b683cabfba643d; no voters:
I20260620 08:07:37.389937 16140 raft_consensus.cc:2781] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:07:37.390007 16140 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:07:37.390038 16140 raft_consensus.cc:3037] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:37.390926 16140 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.391067 16140 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [CANDIDATE]: Term 1 election: Requested vote from peers 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175), 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363)
I20260620 08:07:37.391240 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e693fd10f8144fb29c2e0c9ceff262c2" candidate_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643fa26c0e624a6599b683cabfba643d"
I20260620 08:07:37.391273 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e693fd10f8144fb29c2e0c9ceff262c2" candidate_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091f6815ebf545af85d800f146073d97"
I20260620 08:07:37.391330 15809 raft_consensus.cc:3037] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:37.391376 16079 raft_consensus.cc:3037] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:07:37.392124 15809 raft_consensus.cc:2454] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f9f9d4ca7684ff1b960d7f259bf2f8d in term 1.
I20260620 08:07:37.392150 16079 raft_consensus.cc:2454] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f9f9d4ca7684ff1b960d7f259bf2f8d in term 1.
I20260620 08:07:37.392289 15883 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f9f9d4ca7684ff1b960d7f259bf2f8d, 643fa26c0e624a6599b683cabfba643d; no voters:
I20260620 08:07:37.392432 16140 raft_consensus.cc:2781] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:07:37.392617 16140 raft_consensus.cc:686] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 LEADER]: Becoming Leader. State: Replica: 3f9f9d4ca7684ff1b960d7f259bf2f8d, State: Running, Role: LEADER
I20260620 08:07:37.392721 16140 consensus_queue.cc:237] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:37.393760 15666 catalog_manager.cc:5495] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194). New cstate: current_term: 1 leader_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } health_report { overall_health: UNKNOWN } } }
W20260620 08:07:37.407953 15994 tablet.cc:2369] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:07:37.413012 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.410550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.31.115.194:37853)
0620 08:07:37.410630 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:37.410633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:37.410640 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:07:37.410680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:37.410681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:37.410700 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:37.410760 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:07:37.410866 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.411329 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.412125 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:37.412349 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:37.412425 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:37.412628 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:07:37.412751 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:07:37.412850 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:37.412869 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 08:07:37.412891 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:07:37.415313 15809 raft_consensus.cc:1261] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Refusing update from remote peer 3f9f9d4ca7684ff1b960d7f259bf2f8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:07:37.415297 16079 raft_consensus.cc:1261] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Refusing update from remote peer 3f9f9d4ca7684ff1b960d7f259bf2f8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:07:37.415742 16140 consensus_queue.cc:1035] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:07:37.415946 16142 consensus_queue.cc:1035] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:07:37.418074 16145 mvcc.cc:205] Tried to move back new op lower bound from 7298837943969943552 to 7298837943881371648. Current Snapshot: MvccSnapshot[applied={T|T < 7298837943969943552}]
I20260620 08:07:37.420249 16148 mvcc.cc:205] Tried to move back new op lower bound from 7298837943969943552 to 7298837943881371648. Current Snapshot: MvccSnapshot[applied={T|T < 7298837943969943552}]
I20260620 08:07:37.420644 15667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:07:37.420742 15667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:07:37.421649 15667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Sending DeleteTablet for 3 replicas of tablet e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:07:37.422145 15789 tablet_service.cc:1514] Processing DeleteTablet for tablet e693fd10f8144fb29c2e0c9ceff262c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:37 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46736
I20260620 08:07:37.422158 15927 tablet_service.cc:1514] Processing DeleteTablet for tablet e693fd10f8144fb29c2e0c9ceff262c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:37 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:59588
I20260620 08:07:37.422298 15864 tablet_replica.cc:330] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: stopping tablet replica
I20260620 08:07:37.422380 15864 raft_consensus.cc:2229] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:07:37.422451 15864 raft_consensus.cc:2258] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:07:37.422492 16059 tablet_service.cc:1514] Processing DeleteTablet for tablet e693fd10f8144fb29c2e0c9ceff262c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:07:37 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:54146
I20260620 08:07:37.422618 16154 tablet_replica.cc:330] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: stopping tablet replica
I20260620 08:07:37.422762 16154 raft_consensus.cc:2229] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:07:37.422806 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15705
I20260620 08:07:37.422873 15864 ts_tablet_manager.cc:1905] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:07:37.423568 16155 tablet_replica.cc:330] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: stopping tablet replica
I20260620 08:07:37.423599 16154 raft_consensus.cc:2258] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:07:37.423708 16155 raft_consensus.cc:2229] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:07:37.423841 16155 raft_consensus.cc:2258] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:07:37.424285 16155 ts_tablet_manager.cc:1905] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:07:37.424420 16154 ts_tablet_manager.cc:1905] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:07:37.426445 16154 ts_tablet_manager.cc:1918] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:07:37.426605 16154 log.cc:1196] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:07:37.426608 16155 ts_tablet_manager.cc:1918] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:07:37.426685 16155 log.cc:1196] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:07:37.427030 16155 ts_tablet_manager.cc:1939] T e693fd10f8144fb29c2e0c9ceff262c2 P 091f6815ebf545af85d800f146073d97: Deleting consensus metadata
I20260620 08:07:37.427083 16154 ts_tablet_manager.cc:1939] T e693fd10f8144fb29c2e0c9ceff262c2 P 3f9f9d4ca7684ff1b960d7f259bf2f8d: Deleting consensus metadata
I20260620 08:07:37.427637 15651 catalog_manager.cc:4841] TS 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): tablet e693fd10f8144fb29c2e0c9ceff262c2 (table test-table [id=91ddf7884c9c41079179f75a74277ca1]) successfully deleted
I20260620 08:07:37.427675 15652 catalog_manager.cc:4841] TS 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): tablet e693fd10f8144fb29c2e0c9ceff262c2 (table test-table [id=91ddf7884c9c41079179f75a74277ca1]) successfully deleted
W20260620 08:07:37.430630 15654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:45363 (error 108)
I20260620 08:07:37.431308 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15859
I20260620 08:07:37.431475 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:37.431286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:45363 (local address 127.0.0.1:46738)
0620 08:07:37.431371 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:37.431439 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 08:07:37.432549 15654 catalog_manager.cc:4584] TS 643fa26c0e624a6599b683cabfba643d (127.31.115.193:45363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e693fd10f8144fb29c2e0c9ceff262c2: Network error: Client connection negotiation failed: client connection to 127.31.115.193:45363: connect: Connection refused (error 111)
I20260620 08:07:37.438187 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15996
I20260620 08:07:37.443105 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 15634
I20260620 08:07:57.450299 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36017
--webserver_interface=127.31.115.254
--webserver_port=38925
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:57.542481 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:57.542707 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:57.542732 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:57.543938 16157 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:07:57.543978 16157 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:07:57.543991 16157 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:07:57.544003 16157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:07:57.544016 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:57.544029 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:57.544041 16157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:07:57.544054 16157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:07:57.545899 16157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:57.546190 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:57.546463 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:57.552032 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:57.552044 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:57.552075 16157 server_base.cc:1034] running on GCE node
W20260620 08:07:57.552037 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:57.552690 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:57.553023 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:57.554224 16157 hybrid_clock.cc:648] HybridClock initialized: now 1781942877554188 us; error 57 us; skew 500 ppm
Jun 20 08:07:57 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:57.562458 16157 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:07:57.563149 16157 webserver.cc:460] Webserver started at http://127.31.115.254:38925/ using document root <none> and password file <none>
I20260620 08:07:57.563483 16157 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:57.563560 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:57.565662 16157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:07:57.566603 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:57.567260 16157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:57.567379 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "0858f2fd1499472d98b92560f321f549"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "ab360d9dac96ba40ad6954fe12f48fac"
server_key_iv: "51d31b22099f17a6e89b19d85aa4e0e9"
server_key_version: "encryptionkey@0"
I20260620 08:07:57.567864 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:57.577255 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:57.579878 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:57.580061 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:57.584674 16157 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:36017
I20260620 08:07:57.584712 16224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:36017 every 8 connection(s)
I20260620 08:07:57.585157 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:07:57.586848 16225 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:07:57.586894 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16157
I20260620 08:07:57.587230 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:45363
--local_ip_for_outbound_sockets=127.31.115.193
--tserver_master_addrs=127.31.115.254:36017
--webserver_port=39475
--webserver_interface=127.31.115.193
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:07:57.587718 16225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap starting.
I20260620 08:07:57.597837 16225 log.cc:824] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:57.600412 16225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:07:57.601022 16225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Bootstrap complete.
I20260620 08:07:57.603600 16225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:57.603870 16225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0858f2fd1499472d98b92560f321f549, State: Initialized, Role: FOLLOWER
I20260620 08:07:57.604063 16225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:57.604158 16225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:07:57.604212 16225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:07:57.604261 16225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 2 FOLLOWER]: Advancing to term 3
I20260620 08:07:57.605867 16225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:57.606007 16225 leader_election.cc:302] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0858f2fd1499472d98b92560f321f549; no voters:
I20260620 08:07:57.606238 16225 leader_election.cc:288] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 08:07:57.606289 16229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 08:07:57.606428 16229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [term 3 LEADER]: Becoming Leader. State: Replica: 0858f2fd1499472d98b92560f321f549, State: Running, Role: LEADER
I20260620 08:07:57.606638 16225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:07:57.606621 16229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } }
I20260620 08:07:57.607029 16230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0858f2fd1499472d98b92560f321f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } } }
I20260620 08:07:57.607048 16231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0858f2fd1499472d98b92560f321f549. Latest consensus state: current_term: 3 leader_uuid: "0858f2fd1499472d98b92560f321f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0858f2fd1499472d98b92560f321f549" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36017 } } }
I20260620 08:07:57.607128 16230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: This master's current role is: LEADER
I20260620 08:07:57.607151 16231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549 [sys.catalog]: This master's current role is: LEADER
I20260620 08:07:57.607350 16235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:07:57.608335 16235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:07:57.608496 16235 catalog_manager.cc:1259] Loaded cluster ID: e082f6d7a3a0455982658fea4d289397
I20260620 08:07:57.608726 16235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:07:57.609547 16235 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:07:57.609787 16235 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Loaded TSK: 0
I20260620 08:07:57.609850 16235 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0858f2fd1499472d98b92560f321f549: Loaded TSK: 1
F20260620 08:07:57.610088 16235 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fba431469ad google::LogMessage::Fail() at ??:0
@ 0x7fba4314988a google::LogMessage::SendToLog() at ??:0
@ 0x7fba431464de google::LogMessage::Flush() at ??:0
@ 0x7fba43147a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fba458b3104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fba458b0d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fba4b0a01cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fba4b0a048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fba4b0a0594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fba4b066c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fba4b070ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fba450140d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fba45009954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fba45e6d6db start_thread at ??:0
@ 0x7fba41c4461f clone at ??:0
W20260620 08:07:57.716606 16227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:57.716836 16227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:57.716871 16227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:57.718560 16227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:57.718624 16227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:57.718657 16227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:07:57.720397 16227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:45363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=39475
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:57.720675 16227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:57.720958 16227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:57.728526 16267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:57.728575 16227 server_base.cc:1034] running on GCE node
W20260620 08:07:57.728538 16266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:57.728713 16269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:57.729008 16227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:57.729314 16227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:57.730564 16227 hybrid_clock.cc:648] HybridClock initialized: now 1781942877730542 us; error 37 us; skew 500 ppm
Jun 20 08:07:57 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:57.733273 16227 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:07:57.733928 16227 webserver.cc:460] Webserver started at http://127.31.115.193:39475/ using document root <none> and password file <none>
I20260620 08:07:57.734210 16227 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:57.734294 16227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:57.735872 16227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:07:57.736853 16276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:57.737238 16227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:57.737363 16227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "643fa26c0e624a6599b683cabfba643d"
format_stamp: "Formatted at 2026-06-20 08:07:15 on dist-test-slave-c4w8"
server_key: "4165884cb36b54941e38d7e4dc2ed399"
server_key_iv: "c1d98e7e8ef26a8866384d0b59a40a4e"
server_key_version: "encryptionkey@0"
I20260620 08:07:57.737845 16227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:57.772214 16227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:57.775113 16227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:57.775403 16227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:57.776063 16227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:57.776718 16283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:07:57.778947 16227 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:07:57.779040 16227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:07:57.779095 16227 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:07:57.779799 16227 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:07:57.779865 16227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:07:57.779937 16283 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Bootstrap starting.
I20260620 08:07:57.782027 16283 log.cc:824] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:07:57.785360 16283 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:07:57.785764 16283 tablet_bootstrap.cc:492] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Bootstrap complete.
I20260620 08:07:57.785935 16283 ts_tablet_manager.cc:1397] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260620 08:07:57.787967 16283 raft_consensus.cc:348] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:57.788467 16283 raft_consensus.cc:729] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643fa26c0e624a6599b683cabfba643d, State: Initialized, Role: FOLLOWER
I20260620 08:07:57.788727 16283 consensus_queue.cc:260] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:57.788985 16227 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:45363
I20260620 08:07:57.789058 16283 ts_tablet_manager.cc:1428] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 08:07:57.789222 16392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:45363 every 8 connection(s)
I20260620 08:07:57.789445 16227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:07:57.794956 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16227
I20260620 08:07:57.795300 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:37853
--local_ip_for_outbound_sockets=127.31.115.194
--tserver_master_addrs=127.31.115.254:36017
--webserver_port=45559
--webserver_interface=127.31.115.194
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:57.850425 16395 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:07:57.790486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:38007)
0620 08:07:57.790785 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:57.790840 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:57.791035 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:57.850188 (+ 59153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":151,"threads_started":1}
W20260620 08:07:57.850656 16393 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:36017 (0 consecutive failures): Network error: Failed to ping master at 127.31.115.254:36017: Client connection negotiation failed: client connection to 127.31.115.254:36017: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:07:57.851022 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:57.850887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:40869)
0620 08:07:57.850955 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:57.851000 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 08:07:57.851444 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:57.851348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:49849)
0620 08:07:57.851402 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:57.851426 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 08:07:57.851684 16393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:07:57.893114 16396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:57.893335 16396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:57.893363 16396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:57.894712 16396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:57.894764 16396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:57.894793 16396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:07:57.896310 16396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:37853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=45559
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:57.896590 16396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:57.896848 16396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:57.902064 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:57.902217 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:57.902295 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:57.902801 16396 server_base.cc:1034] running on GCE node
I20260620 08:07:57.903059 16396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:57.903337 16396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:57.904517 16396 hybrid_clock.cc:648] HybridClock initialized: now 1781942877904475 us; error 54 us; skew 500 ppm
Jun 20 08:07:57 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:57.907240 16396 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:07:57.907724 16396 webserver.cc:460] Webserver started at http://127.31.115.194:45559/ using document root <none> and password file <none>
I20260620 08:07:57.907923 16396 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:57.907984 16396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:57.909488 16396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:57.910302 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:57.910626 16396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:07:57.910715 16396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "bcc72036ca16f9f227a09bf97b99a6eb"
server_key_iv: "f48583299173510390665e4fd0bd4e93"
server_key_version: "encryptionkey@0"
I20260620 08:07:57.911115 16396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:57.926901 16396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:57.929705 16396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:57.929886 16396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:57.930166 16396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:57.930495 16396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:57.930536 16396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:57.930560 16396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:57.930573 16396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:57.937451 16396 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:37853
I20260620 08:07:57.937520 16524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:37853 every 8 connection(s)
I20260620 08:07:57.937901 16396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:07:57.939045 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:57.938658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:49223)
0620 08:07:57.938829 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:57.938901 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
W20260620 08:07:57.939502 16525 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:36017 (0 consecutive failures): Network error: Failed to ping master at 127.31.115.254:36017: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
I20260620 08:07:57.939801 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:57.939695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:35355)
0620 08:07:57.939746 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:57.939773 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 08:07:57.940071 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:57.939985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:57279)
0620 08:07:57.940028 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:57.940050 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 08:07:57.940181 16525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:07:57.942793 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16396
I20260620 08:07:57.943133 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:41175
--local_ip_for_outbound_sockets=127.31.115.195
--tserver_master_addrs=127.31.115.254:36017
--webserver_port=35115
--webserver_interface=127.31.115.195
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:07:58.035308 16528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:07:58.035578 16528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:07:58.035617 16528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:07:58.036834 16528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:07:58.036916 16528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:07:58.036963 16528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:07:58.038544 16528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:41175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=35115
--tserver_master_addrs=127.31.115.254:36017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:07:58.038828 16528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:07:58.039084 16528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:07:58.044672 16536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:58.044734 16528 server_base.cc:1034] running on GCE node
W20260620 08:07:58.044662 16533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:07:58.044657 16534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:07:58.045100 16528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:07:58.045359 16528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:07:58.046550 16528 hybrid_clock.cc:648] HybridClock initialized: now 1781942878046518 us; error 39 us; skew 500 ppm
Jun 20 08:07:58 dist-test-slave-c4w8 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:07:58.049392 16528 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:07:58.049963 16528 webserver.cc:460] Webserver started at http://127.31.115.195:35115/ using document root <none> and password file <none>
I20260620 08:07:58.050184 16528 fs_manager.cc:359] Metadata directory not provided
I20260620 08:07:58.050249 16528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:07:58.051787 16528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:07:58.052424 16543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:58.052645 16528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:07:58.052724 16528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "091f6815ebf545af85d800f146073d97"
format_stamp: "Formatted at 2026-06-20 08:07:16 on dist-test-slave-c4w8"
server_key: "703d2af8d36c181341f024852ea97b6f"
server_key_iv: "cca89616c5de1c1ce1fbfe0eb0c5ed33"
server_key_version: "encryptionkey@0"
I20260620 08:07:58.053081 16528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:07:58.085239 16528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:07:58.087788 16528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:07:58.087913 16528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:07:58.088114 16528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:07:58.088450 16528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:07:58.088512 16528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:58.088558 16528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:07:58.088605 16528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:07:58.094811 16528 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41175
I20260620 08:07:58.094866 16656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41175 every 8 connection(s)
I20260620 08:07:58.095224 16528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:07:58.096437 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:58.095941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:53427)
0620 08:07:58.096156 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:58.096236 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
W20260620 08:07:58.096968 16657 heartbeater.cc:644] Failed to heartbeat to 127.31.115.254:36017 (0 consecutive failures): Network error: Failed to ping master at 127.31.115.254:36017: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
I20260620 08:07:58.097261 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:58.097165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:51555)
0620 08:07:58.097212 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:58.097241 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 08:07:58.097522 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:58.097465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:57117)
0620 08:07:58.097489 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:58.097509 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 08:07:58.097645 16657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:07:58.099455 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16528
I20260620 08:07:58.852600 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:58.852190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:43859)
0620 08:07:58.852486 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:58.852527 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260620 08:07:58.940847 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:58.940604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:46875)
0620 08:07:58.940742 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:58.940794 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260620 08:07:59.098323 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.098049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:36051)
0620 08:07:59.098210 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:59.098258 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260620 08:07:59.327452 16664 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:07:59.327586 16664 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:07:59.328049 16664 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
Jun 20 08:07:59 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.194@KRBTEST.COM
Jun 20 08:07:59 dist-test-slave-c4w8 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.195@KRBTEST.COM
I20260620 08:07:59.340145 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.328288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:37853 (local address 127.31.115.193:56533)
0620 08:07:59.328371 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:59.328384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:59.328449 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:59.329459 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:59.329462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:59.329926 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:59.330389 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:59.330392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.331636 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.331637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:59.332311 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:59.332313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.333136 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.333137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:59.333197 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:59.333895 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:59.333900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:59.335527 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:59.337968 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:59.337979 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:59.337982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:59.338348 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:59.338687 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:59.338688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:59.338688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:59.338849 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:59.339373 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:59.339381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:59.339542 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 08:07:59.339843 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 08:07:59.340047 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":96}
I20260620 08:07:59.340170 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.328441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:59727 (local address 127.31.115.195:41175)
0620 08:07:59.328558 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:59.328568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:59.328605 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:07:59.329512 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:59.329516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:59.329564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:59.329731 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:59.330305 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.331245 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.332418 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.332887 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.332981 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:59.335976 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:59.335980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:59.335980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:59.335993 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:59.338236 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:59.338697 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:59.338701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:59.338701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:59.338819 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:59.339148 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:59.339149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:59.339149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:59.339532 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:59.339639 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:59.339871 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 08:07:59.340029 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":49,"threads_started":1}
I20260620 08:07:59.340128 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.328326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41175 (local address 127.31.115.193:59727)
0620 08:07:59.328497 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:59.328508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:07:59.328568 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:07:59.329741 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:07:59.329742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:07:59.329988 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:07:59.330165 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:59.330171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.331376 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.331384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:59.332313 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:07:59.332316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.333028 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.333029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:07:59.333166 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:59.333839 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:07:59.333849 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:07:59.335861 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:07:59.338334 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:59.338335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:59.338336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:59.338609 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:59.338920 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:07:59.338921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:07:59.338922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:07:59.338993 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:07:59.339625 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:07:59.339627 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:07:59.339679 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:07:59.339755 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:07:59.339986 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260620 08:07:59.340340 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.328389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:56533 (local address 127.31.115.194:37853)
0620 08:07:59.328540 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 08:07:59.328552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:07:59.328588 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:07:59.329190 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:07:59.329194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:07:59.329262 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:07:59.329378 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:07:59.330529 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.331508 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.332551 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:07:59.333042 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:07:59.333163 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:07:59.335711 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:07:59.335716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:07:59.335716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:07:59.335728 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:07:59.337866 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:59.338446 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:59.338450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:59.338451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:59.338576 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:07:59.338938 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:07:59.338940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:07:59.338940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:07:59.339252 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:07:59.339315 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:07:59.339990 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 08:07:59.340134 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":50,"threads_started":1}
W20260620 08:07:59.341424 16277 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
W20260620 08:07:59.341670 16278 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:07:59.341768 16278 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643fa26c0e624a6599b683cabfba643d; no voters: 091f6815ebf545af85d800f146073d97, 3f9f9d4ca7684ff1b960d7f259bf2f8d
I20260620 08:07:59.341892 16664 raft_consensus.cc:2726] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:07:59.853431 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.853191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:43031)
0620 08:07:59.853327 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:59.853382 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260620 08:07:59.941772 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:07:59.941403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:33657)
0620 08:07:59.941586 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:07:59.941714 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260620 08:08:00.099296 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:00.098967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:49425)
0620 08:08:00.099188 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:00.099235 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
I20260620 08:08:00.854269 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:00.854005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:45883)
0620 08:08:00.854164 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:00.854209 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260620 08:08:00.942790 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:00.942467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:34957)
0620 08:08:00.942683 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:00.942736 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260620 08:08:01.100391 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:01.100130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:41013)
0620 08:08:01.100286 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:01.100326 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 08:08:01.395751 16674 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:01.395861 16674 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:08:01.396047 16674 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
W20260620 08:08:01.396585 16277 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
W20260620 08:08:01.396662 16278 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:08:01.396755 16278 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643fa26c0e624a6599b683cabfba643d; no voters: 091f6815ebf545af85d800f146073d97, 3f9f9d4ca7684ff1b960d7f259bf2f8d
I20260620 08:08:01.396857 16674 raft_consensus.cc:2726] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:08:01.855320 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:01.854974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:48315)
0620 08:08:01.855218 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:01.855265 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260620 08:08:01.943706 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:01.943413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:57993)
0620 08:08:01.943588 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:01.943635 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":120,"threads_started":1}
I20260620 08:08:02.101300 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:02.100986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:51037)
0620 08:08:02.101200 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:02.101245 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":126,"threads_started":1}
I20260620 08:08:02.856665 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:02.856259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:47013)
0620 08:08:02.856419 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:02.856582 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260620 08:08:02.944572 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:02.944300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:44285)
0620 08:08:02.944478 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:02.944523 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260620 08:08:03.102203 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:03.101944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:41659)
0620 08:08:03.102103 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:03.102151 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260620 08:08:03.619516 16681 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:03.619622 16681 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:08:03.619860 16681 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
W20260620 08:08:03.620311 16278 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
W20260620 08:08:03.620385 16277 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:08:03.620487 16277 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643fa26c0e624a6599b683cabfba643d; no voters: 091f6815ebf545af85d800f146073d97, 3f9f9d4ca7684ff1b960d7f259bf2f8d
I20260620 08:08:03.620608 16681 raft_consensus.cc:2726] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:08:03.857930 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:03.857480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:45843)
0620 08:08:03.857815 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:03.857870 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":80,"threads_started":1}
I20260620 08:08:03.945537 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:03.945305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:33863)
0620 08:08:03.945442 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:03.945487 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":101,"threads_started":1}
I20260620 08:08:04.103152 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:04.102860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:40567)
0620 08:08:04.103055 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:04.103100 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260620 08:08:04.858764 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:04.858527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:37403)
0620 08:08:04.858663 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:04.858712 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260620 08:08:04.946379 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:04.946112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:59385)
0620 08:08:04.946277 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:04.946322 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260620 08:08:05.103957 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:05.103690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:57533)
0620 08:08:05.103854 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:05.103902 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260620 08:08:05.859608 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:05.859375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:57945)
0620 08:08:05.859512 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:05.859550 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260620 08:08:05.947177 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:05.946902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:36973)
0620 08:08:05.947078 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:05.947120 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260620 08:08:06.104725 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:06.104485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:43873)
0620 08:08:06.104631 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:06.104669 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260620 08:08:06.860423 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:06.860140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:35273)
0620 08:08:06.860338 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:06.860375 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":124,"threads_started":1}
I20260620 08:08:06.947945 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:06.947679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:46335)
0620 08:08:06.947853 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:06.947895 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260620 08:08:07.105582 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:07.105308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:55225)
0620 08:08:07.105482 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:07.105531 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260620 08:08:07.861516 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:07.861082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:52267)
0620 08:08:07.861423 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:07.861468 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":207,"threads_started":1}
I20260620 08:08:07.948906 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:07.948599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:49607)
0620 08:08:07.948802 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:07.948853 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260620 08:08:08.092700 16696 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:08.092808 16696 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:08:08.093024 16696 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
W20260620 08:08:08.093447 16277 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
W20260620 08:08:08.093521 16278 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:08:08.093582 16278 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643fa26c0e624a6599b683cabfba643d; no voters: 091f6815ebf545af85d800f146073d97, 3f9f9d4ca7684ff1b960d7f259bf2f8d
I20260620 08:08:08.093691 16696 raft_consensus.cc:2726] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:08:08.106449 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:08.106226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:50909)
0620 08:08:08.106350 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:08.106394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260620 08:08:08.862370 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:08.862087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:55659)
0620 08:08:08.862263 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:08.862319 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260620 08:08:08.950095 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:08.949770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:38371)
0620 08:08:08.949936 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:08.949989 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260620 08:08:09.107292 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:09.107012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:50883)
0620 08:08:09.107197 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:09.107242 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260620 08:08:09.863250 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:09.862993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:47649)
0620 08:08:09.863154 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:09.863199 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260620 08:08:09.951033 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:09.950726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:55277)
0620 08:08:09.950917 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:09.950972 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260620 08:08:10.108150 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:10.107878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:44279)
0620 08:08:10.108042 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:10.108087 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260620 08:08:10.864127 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:10.863858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:46885)
0620 08:08:10.864034 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:10.864079 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260620 08:08:10.951941 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:10.951662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:59115)
0620 08:08:10.951813 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:10.951888 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260620 08:08:11.108930 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:11.108683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:59795)
0620 08:08:11.108832 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:11.108880 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260620 08:08:11.639990 16707 raft_consensus.cc:482] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:11.640093 16707 raft_consensus.cc:504] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9f9d4ca7684ff1b960d7f259bf2f8d" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 37853 } } peers { permanent_uuid: "091f6815ebf545af85d800f146073d97" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41175 } } peers { permanent_uuid: "643fa26c0e624a6599b683cabfba643d" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 45363 } }
I20260620 08:08:11.640311 16707 leader_election.cc:288] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853), 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175)
W20260620 08:08:11.640733 16278 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f9f9d4ca7684ff1b960d7f259bf2f8d (127.31.115.194:37853): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
W20260620 08:08:11.640885 16277 leader_election.cc:341] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 091f6815ebf545af85d800f146073d97 (127.31.115.195:41175): Not found: Tablet not found: e693fd10f8144fb29c2e0c9ceff262c2
I20260620 08:08:11.640920 16277 leader_election.cc:302] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643fa26c0e624a6599b683cabfba643d; no voters: 091f6815ebf545af85d800f146073d97, 3f9f9d4ca7684ff1b960d7f259bf2f8d
I20260620 08:08:11.641016 16707 raft_consensus.cc:2726] T e693fd10f8144fb29c2e0c9ceff262c2 P 643fa26c0e624a6599b683cabfba643d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:08:11.864988 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:11.864698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:48641)
0620 08:08:11.864863 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:11.864920 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260620 08:08:11.952801 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:11.952502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:51015)
0620 08:08:11.952702 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:11.952749 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260620 08:08:12.110085 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:12.109741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:60979)
0620 08:08:12.109970 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:12.110023 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":145,"threads_started":1}
I20260620 08:08:12.865846 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:12.865572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.193:60483)
0620 08:08:12.865750 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:12.865797 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":120,"threads_started":1}
I20260620 08:08:12.953775 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:12.953389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.194:40223)
0620 08:08:12.953672 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:12.953726 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260620 08:08:13.107389 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 16227
I20260620 08:08:13.110873 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:13.110672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36017 (local address 127.31.115.195:36117)
0620 08:08:13.110792 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:13.110825 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.254:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260620 08:08:13.112972 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 16396
I20260620 08:08:13.117885 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 16528
2026-06-20T08:08:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58811 ms)
[----------] 30 tests from SecurityITest (247556 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:13 dist-test-slave-c4w8 krb5kdc[16716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:13 dist-test-slave-c4w8 krb5kdc[16716](info): set up 2 sockets
Jun 20 08:08:13 dist-test-slave-c4w8 krb5kdc[16716](info): commencing operation
krb5kdc: starting...
W20260620 08:08:14.205845 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:14.333307 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33549
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:14.423772 16729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:14.424065 16729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:14.424108 16729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:14.425482 16729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:14.425565 16729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:14.425590 16729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:14.425648 16729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:14.425689 16729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:14.427345 16729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:33549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:14.427650 16729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:14.427901 16729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:14.433586 16734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:14.433566 16737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:14.433561 16735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:14.433676 16729 server_base.cc:1034] running on GCE node
I20260620 08:08:14.433979 16729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:14.434247 16729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:14.435449 16729 hybrid_clock.cc:648] HybridClock initialized: now 1781942894435434 us; error 31 us; skew 500 ppm
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:14.443625 16729 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:14.444175 16729 webserver.cc:460] Webserver started at http://127.31.115.254:38211/ using document root <none> and password file <none>
I20260620 08:08:14.444446 16729 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:14.444495 16729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:14.444634 16729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:14.445818 16729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "f397ecc14be2d0d59457c187fe028534"
server_key_iv: "6495219c1efe606436f74f26db2a0f92"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.446173 16729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "f397ecc14be2d0d59457c187fe028534"
server_key_iv: "6495219c1efe606436f74f26db2a0f92"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.447656 16729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:14.448586 16744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.448784 16729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:14.448874 16729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "f397ecc14be2d0d59457c187fe028534"
server_key_iv: "6495219c1efe606436f74f26db2a0f92"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.448951 16729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:14.473567 16729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:14.475792 16729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:14.475935 16729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:14.480161 16729 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:33549
I20260620 08:08:14.480201 16796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:33549 every 8 connection(s)
I20260620 08:08:14.480587 16729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:14.481644 16797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:14.483937 16797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d: Bootstrap starting.
I20260620 08:08:14.484498 16797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:14.484740 16797 log.cc:824] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:14.485481 16797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d: No bootstrap required, opened a new log
I20260620 08:08:14.487124 16797 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33549 } }
I20260620 08:08:14.487279 16797 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:14.487304 16797 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ffd5ac7a194e8bb716d85a7ac3b40d, State: Initialized, Role: FOLLOWER
I20260620 08:08:14.487439 16797 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33549 } }
I20260620 08:08:14.487522 16797 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:14.487560 16797 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:14.487629 16797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:14.488181 16797 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33549 } }
I20260620 08:08:14.488308 16797 leader_election.cc:302] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ffd5ac7a194e8bb716d85a7ac3b40d; no voters:
I20260620 08:08:14.488492 16797 leader_election.cc:288] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:14.488569 16800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:14.488721 16800 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [term 1 LEADER]: Becoming Leader. State: Replica: d8ffd5ac7a194e8bb716d85a7ac3b40d, State: Running, Role: LEADER
I20260620 08:08:14.488765 16797 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:14.488811 16800 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33549 } }
I20260620 08:08:14.489332 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16729
I20260620 08:08:14.489338 16802 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ffd5ac7a194e8bb716d85a7ac3b40d. Latest consensus state: current_term: 1 leader_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33549 } } }
I20260620 08:08:14.489419 16802 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:14.489444 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:14.489503 16801 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffd5ac7a194e8bb716d85a7ac3b40d" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33549 } } }
I20260620 08:08:14.489576 16801 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:14.489684 32207 external_mini_cluster.cc:1428] Setting key d9bdc6eb61c8faffbe7debadd428af1e
I20260620 08:08:14.489984 16804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:14.490636 16804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:14.492172 16804 catalog_manager.cc:1347] Generated new cluster ID: 4ce47bc249774aea925ae46d945b0d2e
I20260620 08:08:14.492225 16804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:14.499851 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:14.490350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.31.115.254:33549)
0620 08:08:14.490502 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:14.490511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:14.490542 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:08:14.490660 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:14.490668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:14.490727 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:14.491069 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:14.491677 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.492748 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.493507 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.494010 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.494170 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:14.496465 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:14.496470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:14.496470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:14.496484 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:14.498252 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:14.498612 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:14.498616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:14.498616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:14.498718 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:14.498933 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:14.498934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:14.498934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:14.499184 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:14.499354 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:14.499461 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 08:08:14.499604 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260620 08:08:14.500489 16804 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:14.501012 16804 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:14.511885 16804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d: Generated new TSK 0
I20260620 08:08:14.512108 16804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:08:14.550035 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33549
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:14.639257 16823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:14.639482 16823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:14.639508 16823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:14.640717 16823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:14.640765 16823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:14.640793 16823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:08:14.642324 16823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:14.642606 16823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:14.642854 16823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:14.648368 16829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:14.648353 16831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:14.648353 16828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:14.648639 16823 server_base.cc:1034] running on GCE node
I20260620 08:08:14.648958 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:14.649176 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:14.650411 16823 hybrid_clock.cc:648] HybridClock initialized: now 1781942894650380 us; error 49 us; skew 500 ppm
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:14.653260 16823 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:08:14.653806 16823 webserver.cc:460] Webserver started at http://127.31.115.193:34961/ using document root <none> and password file <none>
I20260620 08:08:14.654050 16823 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:14.654098 16823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:14.654235 16823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:14.655306 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "cd31ec4036084b98a1d64442af748eaf"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "c0205f8b6700e76b6a8f3f2862c9292b"
server_key_iv: "73c2c269420e3f7f6c43185b9ee1d259"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.655629 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "cd31ec4036084b98a1d64442af748eaf"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "c0205f8b6700e76b6a8f3f2862c9292b"
server_key_iv: "73c2c269420e3f7f6c43185b9ee1d259"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.657176 16823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:14.658290 16838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.658573 16823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:08:14.658705 16823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "cd31ec4036084b98a1d64442af748eaf"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "c0205f8b6700e76b6a8f3f2862c9292b"
server_key_iv: "73c2c269420e3f7f6c43185b9ee1d259"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.658802 16823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:14.698287 16823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:14.701066 16823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:14.701246 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:14.701514 16823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:14.701853 16823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:14.701902 16823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.701938 16823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:14.701978 16823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.708640 16823 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:33793
I20260620 08:08:14.708685 16951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:33793 every 8 connection(s)
I20260620 08:08:14.709021 16823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:14.717130 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16823
I20260620 08:08:14.717262 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:08:14.717499 32207 external_mini_cluster.cc:1428] Setting key ea0a75a14d2acd4140a5150248e30301
I20260620 08:08:14.720944 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:14.709939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:33911 (local address 127.31.115.254:33549)
0620 08:08:14.710006 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:14.710008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:14.710272 (+ 264us) server_negotiation.cc:407] Connection header received
0620 08:08:14.710773 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:14.710775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:14.710795 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:14.710894 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:14.712256 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.712871 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.714043 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.714270 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.714339 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:14.717276 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:14.717279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:14.717279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:14.717290 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:14.719096 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:14.719752 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:14.719754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:14.719754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:14.719853 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:14.720178 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:14.720179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:14.720180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:14.720349 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:14.720428 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:14.720741 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 08:08:14.720850 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:08:14.721184 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:14.709897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33549 (local address 127.31.115.193:33911)
0620 08:08:14.710068 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:14.710116 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:14.710607 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:14.711012 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:14.711017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:14.711731 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:14.711973 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:14.711979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.712982 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.712983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:14.713933 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:14.713935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.714372 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.714373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:14.714516 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:14.715221 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:14.715237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:14.717081 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:14.719224 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:14.719234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:14.719238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:14.719628 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:14.719969 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:14.719970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:14.719970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:14.720103 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:14.720440 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:14.720447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:14.720585 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 08:08:14.720696 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:08:14.720895 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260620 08:08:14.721768 16952 heartbeater.cc:347] Connected to a master server at 127.31.115.254:33549
I20260620 08:08:14.721896 16952 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:14.722090 16952 heartbeater.cc:511] Master 127.31.115.254:33549 requested a full tablet report, sending...
I20260620 08:08:14.722729 16761 ts_manager.cc:194] Registered new tserver with Master: cd31ec4036084b98a1d64442af748eaf (127.31.115.193:33793)
I20260620 08:08:14.723368 16761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:33911
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:08:14.746842 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33549
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:14.834383 16957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:14.834604 16957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:14.834630 16957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:14.835866 16957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:14.835923 16957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:14.835952 16957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:08:14.837529 16957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:14.837858 16957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:14.838114 16957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:14.843547 16962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:14.843547 16965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:14.843585 16963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:14.843909 16957 server_base.cc:1034] running on GCE node
I20260620 08:08:14.844107 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:14.844388 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:14.845578 16957 hybrid_clock.cc:648] HybridClock initialized: now 1781942894845556 us; error 35 us; skew 500 ppm
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:14.848551 16957 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:08:14.848973 16957 webserver.cc:460] Webserver started at http://127.31.115.194:35607/ using document root <none> and password file <none>
I20260620 08:08:14.849185 16957 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:14.849229 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:14.849323 16957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:14.850435 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "27ff975e9ce9433ebaa1edc840f0775e"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "df62f96fb670b3d8bc590275895a45e2"
server_key_iv: "65fd949e382ed4c5c480954b40d856fe"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.850750 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "27ff975e9ce9433ebaa1edc840f0775e"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "df62f96fb670b3d8bc590275895a45e2"
server_key_iv: "65fd949e382ed4c5c480954b40d856fe"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.852320 16957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:08:14.853145 16972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.853470 16957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:08:14.853590 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "27ff975e9ce9433ebaa1edc840f0775e"
format_stamp: "Formatted at 2026-06-20 08:08:14 on dist-test-slave-c4w8"
server_key: "df62f96fb670b3d8bc590275895a45e2"
server_key_iv: "65fd949e382ed4c5c480954b40d856fe"
server_key_version: "encryptionkey@0"
I20260620 08:08:14.853704 16957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:14.881428 16957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:14.884385 16957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:14.884526 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:14.884876 16957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:14.885239 16957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:14.885279 16957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.885306 16957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:14.885321 16957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:14.892098 16957 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:45841
I20260620 08:08:14.892197 17085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:45841 every 8 connection(s)
I20260620 08:08:14.892604 16957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:08:14.893002 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 16957
I20260620 08:08:14.893133 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:08:14.893391 32207 external_mini_cluster.cc:1428] Setting key f548d3459c5a99f29673285fa3706fc8
Jun 20 08:08:14 dist-test-slave-c4w8 krb5kdc[16716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942894, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:08:14.904232 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:14.893520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:46089 (local address 127.31.115.254:33549)
0620 08:08:14.893581 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:14.893583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:14.893961 (+ 378us) server_negotiation.cc:407] Connection header received
0620 08:08:14.894718 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:14.894719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:14.894731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:14.894791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:14.896179 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.896682 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.897736 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.897950 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.898037 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:14.900813 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:14.900816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:14.900816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:14.900831 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:14.902503 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:14.903005 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:14.903006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:14.903006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:14.903059 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:14.903377 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:14.903378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:14.903379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:14.903519 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:14.903665 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:14.904069 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 08:08:14.904159 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:08:14.904417 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:14.893482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33549 (local address 127.31.115.194:46089)
0620 08:08:14.893796 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:14.893850 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:14.894550 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:14.894904 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:14.894909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:14.895767 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:14.896080 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:14.896085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.896791 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.896792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:14.897637 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:14.897640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:14.898052 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:14.898053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:14.898202 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:14.898794 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:14.898803 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:14.900694 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:14.902597 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:14.902609 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:14.902612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:14.902868 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:14.903137 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:14.903138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:14.903138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:14.903299 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:14.903669 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:14.903677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:14.903817 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 08:08:14.903937 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 08:08:14.904156 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260620 08:08:14.904976 17086 heartbeater.cc:347] Connected to a master server at 127.31.115.254:33549
I20260620 08:08:14.905076 17086 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:14.905274 17086 heartbeater.cc:511] Master 127.31.115.254:33549 requested a full tablet report, sending...
I20260620 08:08:14.905736 16761 ts_manager.cc:194] Registered new tserver with Master: 27ff975e9ce9433ebaa1edc840f0775e (127.31.115.194:45841)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
I20260620 08:08:14.906291 16761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:46089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:08:14.926672 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33549
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:15.012338 17091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:15.012650 17091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:15.012679 17091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:15.014180 17091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:15.014236 17091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:15.014268 17091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:08:15.016595 17091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:15.016937 17091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:15.017216 17091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:15.022732 17096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.022732 17097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.022971 17099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:15.023656 17091 server_base.cc:1034] running on GCE node
I20260620 08:08:15.023898 17091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:15.024130 17091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:15.025321 17091 hybrid_clock.cc:648] HybridClock initialized: now 1781942895025308 us; error 40 us; skew 500 ppm
Jun 20 08:08:15 dist-test-slave-c4w8 krb5kdc[16716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942895, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:15.028283 17091 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:08:15.028913 17091 webserver.cc:460] Webserver started at http://127.31.115.195:45593/ using document root <none> and password file <none>
I20260620 08:08:15.029191 17091 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:15.029260 17091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:15.029414 17091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:15.030591 17091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "a4af18a5611c4059811ade60f77289a5"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "af27e916675510125e0e3ef3e7e4f9d6"
server_key_iv: "fed053a3a5d98b964bb1d51851909ee8"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.030974 17091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "a4af18a5611c4059811ade60f77289a5"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "af27e916675510125e0e3ef3e7e4f9d6"
server_key_iv: "fed053a3a5d98b964bb1d51851909ee8"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.032696 17091 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 08:08:15.033521 17106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.033742 17091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:15.033870 17091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "a4af18a5611c4059811ade60f77289a5"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "af27e916675510125e0e3ef3e7e4f9d6"
server_key_iv: "fed053a3a5d98b964bb1d51851909ee8"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.033957 17091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:15.059607 17091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:15.062338 17091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:15.062543 17091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:15.062801 17091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:15.063164 17091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:15.063231 17091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.063288 17091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:15.063321 17091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.069657 17091 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:36139
I20260620 08:08:15.069712 17219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:36139 every 8 connection(s)
I20260620 08:08:15.070080 17091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:08:15.072651 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17091
I20260620 08:08:15.072772 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:08:15.072950 32207 external_mini_cluster.cc:1428] Setting key 850dc33c4d7f3a38742414d9cdced3fc
Jun 20 08:08:15 dist-test-slave-c4w8 krb5kdc[16716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942895, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:15.080637 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.070886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:53143 (local address 127.31.115.254:33549)
0620 08:08:15.071012 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.071015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.071191 (+ 176us) server_negotiation.cc:407] Connection header received
0620 08:08:15.071822 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.071824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.071838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.071896 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:15.073041 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.073816 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.074838 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.075035 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.075200 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.077599 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:15.077600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:15.077600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:15.077670 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:15.079197 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:15.079633 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:15.079633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:15.079634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:15.079686 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:15.079970 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:15.079971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:15.079971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:15.080084 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:15.080139 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.080498 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.080569 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 08:08:15.080941 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.070844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:33549 (local address 127.31.115.195:53143)
0620 08:08:15.071046 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.071102 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.071654 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.071989 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.071993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.072649 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:15.072941 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.072947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.073909 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.073910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.074743 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.074746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.075213 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.075214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.075322 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.075896 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:15.075904 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:15.077486 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:15.079324 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:15.079332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:15.079334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:15.079561 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:15.079774 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:15.079775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:15.079775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:15.079893 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:15.080165 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:15.080172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:15.080360 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.080469 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.080612 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260620 08:08:15.081545 17220 heartbeater.cc:347] Connected to a master server at 127.31.115.254:33549
I20260620 08:08:15.081693 17220 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:15.081890 17220 heartbeater.cc:511] Master 127.31.115.254:33549 requested a full tablet report, sending...
I20260620 08:08:15.082285 16761 ts_manager.cc:194] Registered new tserver with Master: a4af18a5611c4059811ade60f77289a5 (127.31.115.195:36139)
I20260620 08:08:15.082686 16761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:53143
I20260620 08:08:15.084905 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:08:15.092676 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.086318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.31.115.254:33549)
0620 08:08:15.086393 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.086395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.086478 (+ 83us) server_negotiation.cc:407] Connection header received
0620 08:08:15.086571 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.086572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.086582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.086628 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:15.087187 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.087684 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.088488 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.088720 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.088778 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.090109 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:15.090110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:15.090111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:15.090119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:15.091385 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:15.091850 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:15.091851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:15.091852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:15.091907 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:15.092099 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:15.092100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:15.092100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:15.092265 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:15.092372 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.092484 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.092570 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:08:15.094594 16761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:08:15.096453 16761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:08:15.103607 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.099423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.31.115.193:33793)
0620 08:08:15.100250 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.100269 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.100280 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:15.100317 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.100319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.100338 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.100420 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:15.100528 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.101557 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.102714 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.103267 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.103340 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.103392 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.103488 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.103527 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":77,"threads_started":1}
I20260620 08:08:15.103598 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.099112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:45841 (local address 127.0.0.1:48810)
0620 08:08:15.099578 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.099590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.099643 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.099883 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.099886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.099889 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:15.099978 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.099981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.101086 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.101087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.102291 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.102294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.103231 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.103232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.103322 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.103340 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.103420 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.103484 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260620 08:08:15.103621 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.099112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33793 (local address 127.0.0.1:47744)
0620 08:08:15.099544 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.099554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.099594 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.100417 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.100418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.100420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:15.100454 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.100455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.101711 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.101712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.102622 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.102625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.103340 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.103341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.103404 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.103408 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.103473 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.103524 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260620 08:08:15.103910 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.099410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.31.115.194:45841)
0620 08:08:15.099639 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.099644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.099655 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:15.099704 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.099706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.099718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.099798 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:15.100077 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.100976 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.102402 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.103133 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.103210 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.103268 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.103808 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.103847 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
I20260620 08:08:15.104382 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.099104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:36139 (local address 127.0.0.1:39864)
0620 08:08:15.099290 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.099346 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.099429 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.100173 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.100174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.100176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:15.100221 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.100223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.101524 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.101525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.102635 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.102638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.103625 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.103626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.104215 (+ 589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.104221 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.104279 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.104318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260620 08:08:15.104662 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.099405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.31.115.195:36139)
0620 08:08:15.099952 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.099956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.099965 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:08:15.100004 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.100006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.100035 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.100112 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:15.100294 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.101419 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.102756 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.103482 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.103543 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.103591 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.104535 (+ 944us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.104580 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":87,"threads_started":1}
I20260620 08:08:15.104656 17020 tablet_service.cc:1467] Processing CreateTablet for tablet 2699f4b85b7a4b738a1bcec2c465a948 (DEFAULT_TABLE table=test-table [id=937aaa11b0ed4127bb45ecb25fc77e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:15.104651 16886 tablet_service.cc:1467] Processing CreateTablet for tablet 2699f4b85b7a4b738a1bcec2c465a948 (DEFAULT_TABLE table=test-table [id=937aaa11b0ed4127bb45ecb25fc77e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:15.105036 16886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2699f4b85b7a4b738a1bcec2c465a948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:15.105039 17020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2699f4b85b7a4b738a1bcec2c465a948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:15.105239 17154 tablet_service.cc:1467] Processing CreateTablet for tablet 2699f4b85b7a4b738a1bcec2c465a948 (DEFAULT_TABLE table=test-table [id=937aaa11b0ed4127bb45ecb25fc77e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:15.105530 17154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2699f4b85b7a4b738a1bcec2c465a948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:15.107990 17237 tablet_bootstrap.cc:492] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Bootstrap starting.
I20260620 08:08:15.108675 17238 tablet_bootstrap.cc:492] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Bootstrap starting.
I20260620 08:08:15.108714 17239 tablet_bootstrap.cc:492] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Bootstrap starting.
I20260620 08:08:15.108917 17237 tablet_bootstrap.cc:654] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:15.109333 17239 tablet_bootstrap.cc:654] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:15.109408 17238 tablet_bootstrap.cc:654] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:15.109555 17237 log.cc:824] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:15.109952 17239 log.cc:824] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:15.109997 17238 log.cc:824] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:15.110461 17237 tablet_bootstrap.cc:492] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: No bootstrap required, opened a new log
I20260620 08:08:15.110539 17237 ts_tablet_manager.cc:1397] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:08:15.110733 17239 tablet_bootstrap.cc:492] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: No bootstrap required, opened a new log
I20260620 08:08:15.110823 17239 ts_tablet_manager.cc:1397] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:08:15.112025 17237 raft_consensus.cc:348] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.112205 17237 raft_consensus.cc:374] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:15.112251 17237 raft_consensus.cc:729] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd31ec4036084b98a1d64442af748eaf, State: Initialized, Role: FOLLOWER
I20260620 08:08:15.112375 17237 consensus_queue.cc:260] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.112673 17237 ts_tablet_manager.cc:1428] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:08:15.112747 16952 heartbeater.cc:503] Master 127.31.115.254:33549 was elected leader, sending a full tablet report...
I20260620 08:08:15.112691 17239 raft_consensus.cc:348] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.112870 17239 raft_consensus.cc:374] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:15.112913 17239 raft_consensus.cc:729] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4af18a5611c4059811ade60f77289a5, State: Initialized, Role: FOLLOWER
I20260620 08:08:15.113063 17239 consensus_queue.cc:260] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.113380 17239 ts_tablet_manager.cc:1428] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:08:15.113401 17220 heartbeater.cc:503] Master 127.31.115.254:33549 was elected leader, sending a full tablet report...
I20260620 08:08:15.113870 17238 tablet_bootstrap.cc:492] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: No bootstrap required, opened a new log
I20260620 08:08:15.113945 17238 ts_tablet_manager.cc:1397] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:08:15.115689 17238 raft_consensus.cc:348] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.115824 17238 raft_consensus.cc:374] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:15.115849 17238 raft_consensus.cc:729] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ff975e9ce9433ebaa1edc840f0775e, State: Initialized, Role: FOLLOWER
I20260620 08:08:15.115955 17238 consensus_queue.cc:260] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.116197 17238 ts_tablet_manager.cc:1428] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:08:15.116245 17086 heartbeater.cc:503] Master 127.31.115.254:33549 was elected leader, sending a full tablet report...
W20260620 08:08:15.143337 17087 tablet.cc:2369] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:08:15.209995 16953 tablet.cc:2369] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:08:15.320801 17221 tablet.cc:2369] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:08:15.393208 17245 raft_consensus.cc:482] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:15.393342 17245 raft_consensus.cc:504] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.393863 17245 leader_election.cc:288] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4af18a5611c4059811ade60f77289a5 (127.31.115.195:36139), cd31ec4036084b98a1d64442af748eaf (127.31.115.193:33793)
I20260620 08:08:15.398187 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.394202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:53797 (local address 127.31.115.193:33793)
0620 08:08:15.394290 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.394292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.394301 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:08:15.394344 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.394345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.394354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.394409 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:15.394629 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.395408 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.397083 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.397695 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.397785 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.397797 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.398046 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.398115 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:08:15.398182 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.394246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:56591 (local address 127.31.115.195:36139)
0620 08:08:15.394312 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.394314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.394384 (+ 70us) server_negotiation.cc:407] Connection header received
0620 08:08:15.394478 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.394479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.394487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.394531 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:15.394692 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.395423 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.397083 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.397741 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.397810 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.397819 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.398046 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.398102 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:08:15.398205 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.394072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33793 (local address 127.31.115.194:53797)
0620 08:08:15.394132 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.394144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.394197 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.394440 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.394442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.394444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:15.394535 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.394537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.395539 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.395540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.396955 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.396958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.397832 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.397833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.397941 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.397952 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.398030 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.398088 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":112}
I20260620 08:08:15.398280 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.394077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:36139 (local address 127.31.115.194:56591)
0620 08:08:15.394320 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.394331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.394418 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.394549 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.394550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.394551 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:15.394620 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.394621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.395539 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.395540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.396955 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.396958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.397832 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.397833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.397941 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.397953 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.398030 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.398088 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":127,"thread_start_us":75,"threads_started":1}
I20260620 08:08:15.398674 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2699f4b85b7a4b738a1bcec2c465a948" candidate_uuid: "27ff975e9ce9433ebaa1edc840f0775e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd31ec4036084b98a1d64442af748eaf" is_pre_election: true
I20260620 08:08:15.398684 17174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2699f4b85b7a4b738a1bcec2c465a948" candidate_uuid: "27ff975e9ce9433ebaa1edc840f0775e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4af18a5611c4059811ade60f77289a5" is_pre_election: true
I20260620 08:08:15.398857 17174 raft_consensus.cc:2454] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ff975e9ce9433ebaa1edc840f0775e in term 0.
I20260620 08:08:15.398859 16906 raft_consensus.cc:2454] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ff975e9ce9433ebaa1edc840f0775e in term 0.
I20260620 08:08:15.399128 16973 leader_election.cc:302] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ff975e9ce9433ebaa1edc840f0775e, cd31ec4036084b98a1d64442af748eaf; no voters:
I20260620 08:08:15.399299 17245 raft_consensus.cc:2781] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:08:15.399364 17245 raft_consensus.cc:482] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:08:15.399391 17245 raft_consensus.cc:3037] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:15.400177 17245 raft_consensus.cc:504] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.400369 17245 leader_election.cc:288] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [CANDIDATE]: Term 1 election: Requested vote from peers a4af18a5611c4059811ade60f77289a5 (127.31.115.195:36139), cd31ec4036084b98a1d64442af748eaf (127.31.115.193:33793)
I20260620 08:08:15.400527 17174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2699f4b85b7a4b738a1bcec2c465a948" candidate_uuid: "27ff975e9ce9433ebaa1edc840f0775e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4af18a5611c4059811ade60f77289a5"
I20260620 08:08:15.400630 17174 raft_consensus.cc:3037] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:15.400739 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2699f4b85b7a4b738a1bcec2c465a948" candidate_uuid: "27ff975e9ce9433ebaa1edc840f0775e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd31ec4036084b98a1d64442af748eaf"
I20260620 08:08:15.400837 16906 raft_consensus.cc:3037] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:15.401433 17174 raft_consensus.cc:2454] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ff975e9ce9433ebaa1edc840f0775e in term 1.
I20260620 08:08:15.401687 16976 leader_election.cc:302] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ff975e9ce9433ebaa1edc840f0775e, a4af18a5611c4059811ade60f77289a5; no voters:
I20260620 08:08:15.401732 16906 raft_consensus.cc:2454] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ff975e9ce9433ebaa1edc840f0775e in term 1.
I20260620 08:08:15.401857 17245 raft_consensus.cc:2781] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:15.402065 17245 raft_consensus.cc:686] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 1 LEADER]: Becoming Leader. State: Replica: 27ff975e9ce9433ebaa1edc840f0775e, State: Running, Role: LEADER
I20260620 08:08:15.402177 17245 consensus_queue.cc:237] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } }
I20260620 08:08:15.403080 16761 catalog_manager.cc:5495] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e reported cstate change: term changed from 0 to 1, leader changed from <none> to 27ff975e9ce9433ebaa1edc840f0775e (127.31.115.194). New cstate: current_term: 1 leader_uuid: "27ff975e9ce9433ebaa1edc840f0775e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ff975e9ce9433ebaa1edc840f0775e" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 } health_report { overall_health: UNKNOWN } } }
I20260620 08:08:15.460129 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.455894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.31.115.194:45841)
0620 08:08:15.455973 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.455976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.456049 (+ 73us) server_negotiation.cc:407] Connection header received
0620 08:08:15.456165 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.456167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.456187 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.456298 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:15.456562 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.457627 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.458736 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.458980 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.459090 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.459546 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:15.459680 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:15.459769 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.459971 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.460015 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:08:15.462517 17174 raft_consensus.cc:1261] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 1 FOLLOWER]: Refusing update from remote peer 27ff975e9ce9433ebaa1edc840f0775e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:15.462659 16906 raft_consensus.cc:1261] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 1 FOLLOWER]: Refusing update from remote peer 27ff975e9ce9433ebaa1edc840f0775e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:15.462980 17247 consensus_queue.cc:1035] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4af18a5611c4059811ade60f77289a5" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 36139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:15.463124 17245 consensus_queue.cc:1035] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd31ec4036084b98a1d64442af748eaf" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:15.465921 17251 mvcc.cc:205] Tried to move back new op lower bound from 7298838099811000320 to 7298838099568119808. Current Snapshot: MvccSnapshot[applied={T|T < 7298838099811000320}]
I20260620 08:08:15.467178 17253 mvcc.cc:205] Tried to move back new op lower bound from 7298838099811000320 to 7298838099568119808. Current Snapshot: MvccSnapshot[applied={T|T < 7298838099811000320}]
I20260620 08:08:15.469152 16761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:15.469270 16761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:15.470096 16761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8ffd5ac7a194e8bb716d85a7ac3b40d: Sending DeleteTablet for 3 replicas of tablet 2699f4b85b7a4b738a1bcec2c465a948
I20260620 08:08:15.470494 17154 tablet_service.cc:1514] Processing DeleteTablet for tablet 2699f4b85b7a4b738a1bcec2c465a948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:15 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:39864
I20260620 08:08:15.470492 16886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2699f4b85b7a4b738a1bcec2c465a948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:15 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:47744
I20260620 08:08:15.470744 17260 tablet_replica.cc:330] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: stopping tablet replica
I20260620 08:08:15.470880 17020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2699f4b85b7a4b738a1bcec2c465a948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:15 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:48810
I20260620 08:08:15.470929 17260 raft_consensus.cc:2229] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:15.471055 17260 raft_consensus.cc:2258] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:15.471117 17261 tablet_replica.cc:330] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: stopping tablet replica
I20260620 08:08:15.471272 17261 raft_consensus.cc:2229] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:08:15.471333 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 16823
I20260620 08:08:15.471416 17261 raft_consensus.cc:2258] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:15.471603 17260 ts_tablet_manager.cc:1905] T 2699f4b85b7a4b738a1bcec2c465a948 P cd31ec4036084b98a1d64442af748eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:15.471925 17259 tablet_replica.cc:330] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: stopping tablet replica
I20260620 08:08:15.472031 17259 raft_consensus.cc:2229] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:15.472134 17259 raft_consensus.cc:2258] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:15.472586 17259 ts_tablet_manager.cc:1905] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:15.473102 17261 ts_tablet_manager.cc:1905] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:15.474864 17259 ts_tablet_manager.cc:1918] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:15.474898 17261 ts_tablet_manager.cc:1918] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:15.474961 17259 log.cc:1196] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/2699f4b85b7a4b738a1bcec2c465a948
I20260620 08:08:15.475013 17261 log.cc:1196] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/2699f4b85b7a4b738a1bcec2c465a948
I20260620 08:08:15.475286 17259 ts_tablet_manager.cc:1939] T 2699f4b85b7a4b738a1bcec2c465a948 P a4af18a5611c4059811ade60f77289a5: Deleting consensus metadata
I20260620 08:08:15.475334 17261 ts_tablet_manager.cc:1939] T 2699f4b85b7a4b738a1bcec2c465a948 P 27ff975e9ce9433ebaa1edc840f0775e: Deleting consensus metadata
I20260620 08:08:15.476198 16748 catalog_manager.cc:4841] TS a4af18a5611c4059811ade60f77289a5 (127.31.115.195:36139): tablet 2699f4b85b7a4b738a1bcec2c465a948 (table test-table [id=937aaa11b0ed4127bb45ecb25fc77e37]) successfully deleted
I20260620 08:08:15.478070 16747 catalog_manager.cc:4841] TS 27ff975e9ce9433ebaa1edc840f0775e (127.31.115.194:45841): tablet 2699f4b85b7a4b738a1bcec2c465a948 (table test-table [id=937aaa11b0ed4127bb45ecb25fc77e37]) successfully deleted
W20260620 08:08:15.480150 16745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:33793 (error 108)
I20260620 08:08:15.480307 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 16957
I20260620 08:08:15.482220 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.482047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33793 (local address 127.0.0.1:47758)
0620 08:08:15.482113 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.482166 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 08:08:15.482391 16745 catalog_manager.cc:4584] TS cd31ec4036084b98a1d64442af748eaf (127.31.115.193:33793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2699f4b85b7a4b738a1bcec2c465a948: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33793: connect: Connection refused (error 111)
I20260620 08:08:15.488755 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17091
I20260620 08:08:15.494793 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 16729
2026-06-20T08:08:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2378 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T08:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:15Z Disabled control of system clock
I20260620 08:08:15.525028 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:42981
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:15.618328 17266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:15.618571 17266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:15.618594 17266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:15.619891 17266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:15.619941 17266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:15.619958 17266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:15.619972 17266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:15.619984 17266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:15.621428 17266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:42981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:15.621718 17266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:15.622053 17266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:15.627696 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.627696 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:15.627729 17266 server_base.cc:1034] running on GCE node
W20260620 08:08:15.627703 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:15.628191 17266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:15.628553 17266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:15.629709 17266 hybrid_clock.cc:648] HybridClock initialized: now 1781942895629705 us; error 48 us; skew 500 ppm
I20260620 08:08:15.631584 17266 webserver.cc:460] Webserver started at http://127.31.115.254:41907/ using document root <none> and password file <none>
I20260620 08:08:15.631870 17266 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:15.631953 17266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:15.632115 17266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:15.633860 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "1274bf11fa01415187cb447acc65fe84"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "a6c2ef84761bc1dcd8894ff52d979205"
server_key_iv: "16f2997a4a3add4150efb06bcdf4acf3"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.634505 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "1274bf11fa01415187cb447acc65fe84"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "a6c2ef84761bc1dcd8894ff52d979205"
server_key_iv: "16f2997a4a3add4150efb06bcdf4acf3"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.636453 17266 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 08:08:15.637435 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.637715 17266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:15.637831 17266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "1274bf11fa01415187cb447acc65fe84"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "a6c2ef84761bc1dcd8894ff52d979205"
server_key_iv: "16f2997a4a3add4150efb06bcdf4acf3"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.637950 17266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:15.676601 17266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:15.676990 17266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:15.677160 17266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:15.683806 17266 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:42981
I20260620 08:08:15.683877 17332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:42981 every 8 connection(s)
I20260620 08:08:15.684298 17266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:15.685544 17333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:15.688241 17333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84: Bootstrap starting.
I20260620 08:08:15.688862 17333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:15.689124 17333 log.cc:824] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:15.689935 17333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84: No bootstrap required, opened a new log
I20260620 08:08:15.691102 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17266
I20260620 08:08:15.691241 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:15.691506 32207 external_mini_cluster.cc:1428] Setting key 8ce8c5ae5c31ebf6f2a365df07bdb82f
I20260620 08:08:15.691826 17333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1274bf11fa01415187cb447acc65fe84" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42981 } }
I20260620 08:08:15.692106 17333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:15.692174 17333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1274bf11fa01415187cb447acc65fe84, State: Initialized, Role: FOLLOWER
I20260620 08:08:15.692345 17333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1274bf11fa01415187cb447acc65fe84" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42981 } }
I20260620 08:08:15.692452 17333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:15.692484 17333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:15.692531 17333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:15.693331 17333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1274bf11fa01415187cb447acc65fe84" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42981 } }
I20260620 08:08:15.693454 17333 leader_election.cc:302] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1274bf11fa01415187cb447acc65fe84; no voters:
I20260620 08:08:15.693624 17333 leader_election.cc:288] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:15.693706 17338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:15.693885 17333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:15.693923 17338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [term 1 LEADER]: Becoming Leader. State: Replica: 1274bf11fa01415187cb447acc65fe84, State: Running, Role: LEADER
I20260620 08:08:15.694149 17338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1274bf11fa01415187cb447acc65fe84" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42981 } }
I20260620 08:08:15.694555 17339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1274bf11fa01415187cb447acc65fe84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1274bf11fa01415187cb447acc65fe84" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42981 } } }
I20260620 08:08:15.694643 17339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:15.694828 17340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1274bf11fa01415187cb447acc65fe84. Latest consensus state: current_term: 1 leader_uuid: "1274bf11fa01415187cb447acc65fe84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1274bf11fa01415187cb447acc65fe84" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 42981 } } }
I20260620 08:08:15.694859 17347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:15.694888 17340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:15.695355 17347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:15.697167 17347 catalog_manager.cc:1347] Generated new cluster ID: d7190de9b9924f49ab8dea02fc6e695b
I20260620 08:08:15.697252 17347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:15.698256 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.692128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.31.115.254:42981)
0620 08:08:15.692348 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.692357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.692388 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:08:15.692507 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.692515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.692589 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.692726 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:15.692940 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.694803 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.695947 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.696465 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.696589 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.697398 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:15.697402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:15.697402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:15.697641 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:15.697691 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:15.697706 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:15.697759 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.697976 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.698054 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260620 08:08:15.720036 17347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:15.720614 17347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:15.730589 17347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84: Generated new TSK 0
I20260620 08:08:15.730809 17347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:15.736610 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42981
--builtin_ntp_servers=127.31.115.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:15.829274 17357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:15.829523 17357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:15.829548 17357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:15.830860 17357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:15.830919 17357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:15.830946 17357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:08:15.832526 17357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:15.832806 17357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:15.833137 17357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:15.839262 17365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.839280 17363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.839514 17362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:15.839577 17357 server_base.cc:1034] running on GCE node
I20260620 08:08:15.839867 17357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:15.840147 17357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:15.841343 17357 hybrid_clock.cc:648] HybridClock initialized: now 1781942895841314 us; error 42 us; skew 500 ppm
I20260620 08:08:15.842919 17357 webserver.cc:460] Webserver started at http://127.31.115.193:41511/ using document root <none> and password file <none>
I20260620 08:08:15.843211 17357 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:15.843259 17357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:15.843398 17357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:15.844664 17357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "6814a07f8a7640f8818c49636972584c"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "4af6178888cfb4260bcce00acbbfaac4"
server_key_iv: "4ed9a7a2aa2fd9349d12c25ea2cbef75"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.845048 17357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "6814a07f8a7640f8818c49636972584c"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "4af6178888cfb4260bcce00acbbfaac4"
server_key_iv: "4ed9a7a2aa2fd9349d12c25ea2cbef75"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.846665 17357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:15.847664 17371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.847957 17357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:15.848078 17357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "6814a07f8a7640f8818c49636972584c"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "4af6178888cfb4260bcce00acbbfaac4"
server_key_iv: "4ed9a7a2aa2fd9349d12c25ea2cbef75"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.848177 17357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:15.870255 17357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:15.870646 17357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:15.870814 17357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:15.871080 17357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:15.871444 17357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:15.871503 17357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.871547 17357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:15.871592 17357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.878597 17357 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:43791
I20260620 08:08:15.878841 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:43791 every 8 connection(s)
I20260620 08:08:15.879184 17357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:08:15.883208 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17357
I20260620 08:08:15.883309 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:08:15.883559 32207 external_mini_cluster.cc:1428] Setting key 60dc3da2a2e59e0c21e6ca20e19580ee
I20260620 08:08:15.884784 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42981
--builtin_ntp_servers=127.31.115.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:08:15.886462 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.880472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:44155 (local address 127.31.115.254:42981)
0620 08:08:15.880682 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:15.880685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:15.880896 (+ 211us) server_negotiation.cc:407] Connection header received
0620 08:08:15.881582 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:15.881584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:15.881602 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:15.881682 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:15.882420 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.883007 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.884490 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.884800 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.884880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.886125 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:15.886127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:15.886128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:15.886134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:15.886150 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:15.886154 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:15.886223 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:15.886334 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 08:08:15.886376 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260620 08:08:15.886737 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:15.880333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:42981 (local address 127.31.115.193:44155)
0620 08:08:15.880736 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:15.880784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:15.881359 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:15.881770 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:15.881777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:15.881791 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:15.882314 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.882319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.883146 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.883147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.884337 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:15.884341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:15.885013 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:15.885014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:15.885224 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:15.885940 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:15.885941 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:15.885979 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:15.886006 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:15.886239 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:15.886245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:15.886249 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:15.886338 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:08:15.886508 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":102,"threads_started":1}
I20260620 08:08:15.887262 17485 heartbeater.cc:347] Connected to a master server at 127.31.115.254:42981
I20260620 08:08:15.887375 17485 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:15.887598 17485 heartbeater.cc:511] Master 127.31.115.254:42981 requested a full tablet report, sending...
I20260620 08:08:15.888165 17297 ts_manager.cc:194] Registered new tserver with Master: 6814a07f8a7640f8818c49636972584c (127.31.115.193:43791)
I20260620 08:08:15.888855 17297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.193:44155
W20260620 08:08:15.974428 17488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:15.974658 17488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:15.974681 17488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:15.975997 17488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:15.976050 17488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:15.976079 17488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:08:15.977548 17488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:15.977900 17488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:15.978240 17488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:15.984552 17496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.984352 17493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:15.984352 17494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:15.984700 17488 server_base.cc:1034] running on GCE node
I20260620 08:08:15.984947 17488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:15.985241 17488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:15.986423 17488 hybrid_clock.cc:648] HybridClock initialized: now 1781942895986390 us; error 44 us; skew 500 ppm
I20260620 08:08:15.987952 17488 webserver.cc:460] Webserver started at http://127.31.115.194:35009/ using document root <none> and password file <none>
I20260620 08:08:15.988246 17488 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:15.988322 17488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:15.988497 17488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:15.989694 17488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "9a5fddda74b1409490bca07311c9c0fe"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "4dffa0372366a74ef83889a59f12dacc"
server_key_iv: "17a4f4e8027658007184b908fb19842e"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.990077 17488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "9a5fddda74b1409490bca07311c9c0fe"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "4dffa0372366a74ef83889a59f12dacc"
server_key_iv: "17a4f4e8027658007184b908fb19842e"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.991564 17488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:08:15.992491 17502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:15.992764 17488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:15.992887 17488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "9a5fddda74b1409490bca07311c9c0fe"
format_stamp: "Formatted at 2026-06-20 08:08:15 on dist-test-slave-c4w8"
server_key: "4dffa0372366a74ef83889a59f12dacc"
server_key_iv: "17a4f4e8027658007184b908fb19842e"
server_key_version: "encryptionkey@0"
I20260620 08:08:15.992969 17488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:16.003827 17488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:16.004180 17488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:16.004343 17488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:16.004585 17488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:16.004972 17488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:16.005028 17488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:16.005074 17488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:16.005106 17488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:16.012151 17488 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:42221
I20260620 08:08:16.012192 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:42221 every 8 connection(s)
I20260620 08:08:16.012575 17488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:08:16.018431 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.013475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:59231 (local address 127.31.115.254:42981)
0620 08:08:16.013706 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.013708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.013835 (+ 127us) server_negotiation.cc:407] Connection header received
0620 08:08:16.014285 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.014286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.014297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.014373 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:16.015079 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.015592 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.016785 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.017021 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.017083 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.018125 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:16.018126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:16.018126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:16.018131 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:16.018147 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:16.018151 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:16.018212 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.018329 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.018360 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260620 08:08:16.018538 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.013436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:42981 (local address 127.31.115.194:59231)
0620 08:08:16.013657 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.013700 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.014190 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.014428 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.014431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.014441 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:16.014961 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.014965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.015723 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.015725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.016687 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.016691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.017147 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.017149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.017272 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.017982 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:16.017982 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:16.018012 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:16.018038 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:16.018215 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:16.018217 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:16.018220 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.018279 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.018366 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260620 08:08:16.019331 17616 heartbeater.cc:347] Connected to a master server at 127.31.115.254:42981
I20260620 08:08:16.019433 17616 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:16.019590 17616 heartbeater.cc:511] Master 127.31.115.254:42981 requested a full tablet report, sending...
I20260620 08:08:16.020000 17297 ts_manager.cc:194] Registered new tserver with Master: 9a5fddda74b1409490bca07311c9c0fe (127.31.115.194:42221)
I20260620 08:08:16.020346 17297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.194:59231
I20260620 08:08:16.020838 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17488
I20260620 08:08:16.020949 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:08:16.021163 32207 external_mini_cluster.cc:1428] Setting key 67d58a1d094c8d64d212a38fb538f0e6
I20260620 08:08:16.022226 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42981
--builtin_ntp_servers=127.31.115.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:16.110579 17619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:16.110811 17619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:16.110836 17619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:16.111966 17619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:16.112007 17619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:16.112033 17619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:08:16.113343 17619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:42981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:16.113556 17619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:16.113873 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:16.118978 17625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:16.118949 17627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:16.118937 17624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:16.119307 17619 server_base.cc:1034] running on GCE node
I20260620 08:08:16.119500 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:16.119692 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:16.120854 17619 hybrid_clock.cc:648] HybridClock initialized: now 1781942896120837 us; error 29 us; skew 500 ppm
I20260620 08:08:16.122413 17619 webserver.cc:460] Webserver started at http://127.31.115.195:38975/ using document root <none> and password file <none>
I20260620 08:08:16.122682 17619 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:16.122752 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:16.122907 17619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:16.124032 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "f78de6e45bb646f8b6c0f9ae94f92444"
format_stamp: "Formatted at 2026-06-20 08:08:16 on dist-test-slave-c4w8"
server_key: "c2f0f3cebabf958f223a71d735dff1a8"
server_key_iv: "4a5fda967c9ef3a93e97de5da074de13"
server_key_version: "encryptionkey@0"
I20260620 08:08:16.124406 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "f78de6e45bb646f8b6c0f9ae94f92444"
format_stamp: "Formatted at 2026-06-20 08:08:16 on dist-test-slave-c4w8"
server_key: "c2f0f3cebabf958f223a71d735dff1a8"
server_key_iv: "4a5fda967c9ef3a93e97de5da074de13"
server_key_version: "encryptionkey@0"
I20260620 08:08:16.125869 17619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 08:08:16.126788 17633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:16.127034 17619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:16.127173 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "f78de6e45bb646f8b6c0f9ae94f92444"
format_stamp: "Formatted at 2026-06-20 08:08:16 on dist-test-slave-c4w8"
server_key: "c2f0f3cebabf958f223a71d735dff1a8"
server_key_iv: "4a5fda967c9ef3a93e97de5da074de13"
server_key_version: "encryptionkey@0"
I20260620 08:08:16.127270 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:16.136893 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:16.137229 17619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:16.137398 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:16.137844 17619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:16.138227 17619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:16.138298 17619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:16.138370 17619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:16.138417 17619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:16.145068 17619 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:37273
I20260620 08:08:16.145116 17746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:37273 every 8 connection(s)
I20260620 08:08:16.145474 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:08:16.147744 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17619
I20260620 08:08:16.147859 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:08:16.148088 32207 external_mini_cluster.cc:1428] Setting key e8dad9e49095bfa508105bfd1ff5db82
I20260620 08:08:16.151284 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.146267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:41593 (local address 127.31.115.254:42981)
0620 08:08:16.146339 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.146342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.146598 (+ 256us) server_negotiation.cc:407] Connection header received
0620 08:08:16.147087 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.147088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.147099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.147155 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:16.148033 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.148654 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.149871 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.150065 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.150131 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.151019 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:16.151020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:16.151020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:16.151025 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:16.151039 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:16.151043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:16.151093 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.151187 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.151219 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:08:16.151520 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.146275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:42981 (local address 127.31.115.195:41593)
0620 08:08:16.146474 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.146509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.147007 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.147207 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.147210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.147218 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:16.147914 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.147922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.148763 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.148764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.149790 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.149793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.150146 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.150146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.150256 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.150864 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:16.150865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:16.150893 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:16.150917 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:16.151112 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:16.151118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:16.151120 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.151176 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.151301 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260620 08:08:16.152100 17747 heartbeater.cc:347] Connected to a master server at 127.31.115.254:42981
I20260620 08:08:16.152200 17747 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:16.152391 17747 heartbeater.cc:511] Master 127.31.115.254:42981 requested a full tablet report, sending...
I20260620 08:08:16.152871 17297 ts_manager.cc:194] Registered new tserver with Master: f78de6e45bb646f8b6c0f9ae94f92444 (127.31.115.195:37273)
I20260620 08:08:16.153220 17297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.195:41593
I20260620 08:08:16.160043 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:08:16.164870 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.161596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.31.115.254:42981)
0620 08:08:16.161685 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.161688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.161787 (+ 99us) server_negotiation.cc:407] Connection header received
0620 08:08:16.161867 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.161868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.161881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.161930 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:16.162098 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.162602 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.163404 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.163615 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.163707 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.164513 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:16.164515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:16.164515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:16.164520 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:16.164536 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:16.164540 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:16.164619 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.164687 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.164717 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:08:16.166877 17297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:08:16.168614 17297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:08:16.175428 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.171437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:37273 (local address 127.0.0.1:41748)
0620 08:08:16.171725 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.171733 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.171767 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.172062 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.172063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.172065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:16.172123 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.172125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.173000 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.173002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.174110 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.174114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.175086 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.175169 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.175184 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.175272 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.175342 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260620 08:08:16.175434 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.171589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.31.115.195:37273)
0620 08:08:16.171845 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.171849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.171860 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:16.171898 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.171900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.171915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.171989 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:16.172201 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.172905 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.174189 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.174995 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175079 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.175176 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.175288 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.175334 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":167,"threads_started":1}
I20260620 08:08:16.175585 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.171166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:43791 (local address 127.0.0.1:34700)
0620 08:08:16.171295 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.171341 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.171411 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.172019 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.172023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.172027 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:16.172112 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.172116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.173728 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.173729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.174579 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.174582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.175374 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.175432 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.175436 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.175464 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.175504 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":124,"threads_started":1}
I20260620 08:08:16.176043 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.171556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.31.115.193:43791)
0620 08:08:16.171786 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.171792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.171806 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:08:16.171858 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.171860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.171877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.172212 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:16.172284 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.173648 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.174674 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.175272 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175655 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.175749 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.175774 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.175812 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260620 08:08:16.176119 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.171246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.31.115.194:42221)
0620 08:08:16.171884 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.171888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.171907 (+ 19us) server_negotiation.cc:407] Connection header received
0620 08:08:16.171946 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.171949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.171964 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.172215 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:16.172283 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.173629 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175002 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.175726 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175850 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.175936 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.175994 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.176032 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":123,"threads_started":1}
I20260620 08:08:16.176100 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.171166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:42221 (local address 127.0.0.1:33678)
0620 08:08:16.171569 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.171580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.171631 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.172067 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.172068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.172070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:16.172112 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.172115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.173744 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.173745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.174871 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.174875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.175852 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.175853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.175915 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.175919 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.175980 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.176020 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260620 08:08:16.176751 17419 tablet_service.cc:1467] Processing CreateTablet for tablet 674688da098b420a882762b71bb17a90 (DEFAULT_TABLE table=test-table [id=25315729152c48cb816a9bc2a8759ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:16.176756 17681 tablet_service.cc:1467] Processing CreateTablet for tablet 674688da098b420a882762b71bb17a90 (DEFAULT_TABLE table=test-table [id=25315729152c48cb816a9bc2a8759ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:16.177119 17419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674688da098b420a882762b71bb17a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:16.177120 17681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674688da098b420a882762b71bb17a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:16.180320 17761 tablet_bootstrap.cc:492] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: Bootstrap starting.
I20260620 08:08:16.180351 17762 tablet_bootstrap.cc:492] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Bootstrap starting.
I20260620 08:08:16.181085 17761 tablet_bootstrap.cc:654] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:16.181129 17762 tablet_bootstrap.cc:654] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:16.181164 17550 tablet_service.cc:1467] Processing CreateTablet for tablet 674688da098b420a882762b71bb17a90 (DEFAULT_TABLE table=test-table [id=25315729152c48cb816a9bc2a8759ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:16.181465 17762 log.cc:824] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:16.181485 17761 log.cc:824] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:16.181577 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674688da098b420a882762b71bb17a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:16.182250 17762 tablet_bootstrap.cc:492] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: No bootstrap required, opened a new log
I20260620 08:08:16.182256 17761 tablet_bootstrap.cc:492] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: No bootstrap required, opened a new log
I20260620 08:08:16.182328 17762 ts_tablet_manager.cc:1397] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:08:16.182350 17761 ts_tablet_manager.cc:1397] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:08:16.184429 17761 raft_consensus.cc:348] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.184561 17761 raft_consensus.cc:374] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:16.184583 17761 raft_consensus.cc:729] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6814a07f8a7640f8818c49636972584c, State: Initialized, Role: FOLLOWER
I20260620 08:08:16.184719 17761 consensus_queue.cc:260] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.185017 17761 ts_tablet_manager.cc:1428] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:08:16.185055 17485 heartbeater.cc:503] Master 127.31.115.254:42981 was elected leader, sending a full tablet report...
I20260620 08:08:16.185221 17766 tablet_bootstrap.cc:492] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Bootstrap starting.
I20260620 08:08:16.185379 17762 raft_consensus.cc:348] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.185573 17762 raft_consensus.cc:374] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:16.185674 17762 raft_consensus.cc:729] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78de6e45bb646f8b6c0f9ae94f92444, State: Initialized, Role: FOLLOWER
I20260620 08:08:16.185823 17762 consensus_queue.cc:260] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.185946 17766 tablet_bootstrap.cc:654] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:16.186197 17762 ts_tablet_manager.cc:1428] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 08:08:16.186256 17747 heartbeater.cc:503] Master 127.31.115.254:42981 was elected leader, sending a full tablet report...
I20260620 08:08:16.186309 17766 log.cc:824] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:16.187120 17766 tablet_bootstrap.cc:492] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: No bootstrap required, opened a new log
I20260620 08:08:16.187206 17766 ts_tablet_manager.cc:1397] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:08:16.189141 17766 raft_consensus.cc:348] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.189296 17766 raft_consensus.cc:374] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:16.189343 17766 raft_consensus.cc:729] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5fddda74b1409490bca07311c9c0fe, State: Initialized, Role: FOLLOWER
I20260620 08:08:16.189473 17766 consensus_queue.cc:260] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.189764 17766 ts_tablet_manager.cc:1428] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 08:08:16.189821 17616 heartbeater.cc:503] Master 127.31.115.254:42981 was elected leader, sending a full tablet report...
W20260620 08:08:16.263240 17617 tablet.cc:2369] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:08:16.346354 17769 raft_consensus.cc:482] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:16.346484 17769 raft_consensus.cc:504] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.346967 17769 leader_election.cc:288] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6814a07f8a7640f8818c49636972584c (127.31.115.193:43791), f78de6e45bb646f8b6c0f9ae94f92444 (127.31.115.195:37273)
I20260620 08:08:16.351284 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.347263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:34285 (local address 127.31.115.193:43791)
0620 08:08:16.347319 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.347321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.347338 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:08:16.347466 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.347468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.347484 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.347540 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:16.347767 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.348536 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.350214 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.350785 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.350904 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.350914 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.351145 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.351193 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 08:08:16.351298 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.347305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:37273 (local address 127.31.115.194:49905)
0620 08:08:16.347508 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.347519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.347565 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.347872 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.347874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.347876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:16.347965 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.347969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.348894 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.348895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.350077 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.350079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.350944 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.350945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.351049 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.351051 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.351123 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.351214 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260620 08:08:16.351272 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.347478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:49905 (local address 127.31.115.195:37273)
0620 08:08:16.347648 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.347650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.347661 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:16.347697 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.347699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.347712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.347802 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:16.348079 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.348798 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.350212 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.350821 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.350904 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.350914 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.351146 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.351194 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 08:08:16.351388 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.347158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:43791 (local address 127.31.115.194:34285)
0620 08:08:16.347234 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.347248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:16.347358 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:16.347596 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:16.347597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:16.347598 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:16.347678 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.347680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.348653 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.348655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.350024 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:16.350030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.350944 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.350945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:16.351041 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.351046 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:08:16.351132 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:08:16.351182 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":4}
I20260620 08:08:16.351842 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674688da098b420a882762b71bb17a90" candidate_uuid: "9a5fddda74b1409490bca07311c9c0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" is_pre_election: true
I20260620 08:08:16.351866 17439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674688da098b420a882762b71bb17a90" candidate_uuid: "9a5fddda74b1409490bca07311c9c0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6814a07f8a7640f8818c49636972584c" is_pre_election: true
I20260620 08:08:16.352030 17701 raft_consensus.cc:2454] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5fddda74b1409490bca07311c9c0fe in term 0.
I20260620 08:08:16.352077 17439 raft_consensus.cc:2454] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5fddda74b1409490bca07311c9c0fe in term 0.
I20260620 08:08:16.352348 17503 leader_election.cc:302] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5fddda74b1409490bca07311c9c0fe, f78de6e45bb646f8b6c0f9ae94f92444; no voters:
I20260620 08:08:16.352494 17769 raft_consensus.cc:2781] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:08:16.352612 17769 raft_consensus.cc:482] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:08:16.352649 17769 raft_consensus.cc:3037] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:16.353544 17769 raft_consensus.cc:504] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.353724 17769 leader_election.cc:288] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [CANDIDATE]: Term 1 election: Requested vote from peers 6814a07f8a7640f8818c49636972584c (127.31.115.193:43791), f78de6e45bb646f8b6c0f9ae94f92444 (127.31.115.195:37273)
I20260620 08:08:16.353924 17439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674688da098b420a882762b71bb17a90" candidate_uuid: "9a5fddda74b1409490bca07311c9c0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6814a07f8a7640f8818c49636972584c"
I20260620 08:08:16.353976 17701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674688da098b420a882762b71bb17a90" candidate_uuid: "9a5fddda74b1409490bca07311c9c0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f78de6e45bb646f8b6c0f9ae94f92444"
I20260620 08:08:16.354022 17439 raft_consensus.cc:3037] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:16.354095 17701 raft_consensus.cc:3037] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:16.354773 17439 raft_consensus.cc:2454] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5fddda74b1409490bca07311c9c0fe in term 1.
I20260620 08:08:16.354950 17503 leader_election.cc:302] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6814a07f8a7640f8818c49636972584c, 9a5fddda74b1409490bca07311c9c0fe; no voters:
I20260620 08:08:16.355039 17701 raft_consensus.cc:2454] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5fddda74b1409490bca07311c9c0fe in term 1.
I20260620 08:08:16.355084 17769 raft_consensus.cc:2781] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:16.355253 17769 raft_consensus.cc:686] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 1 LEADER]: Becoming Leader. State: Replica: 9a5fddda74b1409490bca07311c9c0fe, State: Running, Role: LEADER
I20260620 08:08:16.355363 17769 consensus_queue.cc:237] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } }
I20260620 08:08:16.356184 17297 catalog_manager.cc:5495] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a5fddda74b1409490bca07311c9c0fe (127.31.115.194). New cstate: current_term: 1 leader_uuid: "9a5fddda74b1409490bca07311c9c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a5fddda74b1409490bca07311c9c0fe" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 42221 } health_report { overall_health: HEALTHY } } }
W20260620 08:08:16.380225 17486 tablet.cc:2369] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:08:16.396703 17748 tablet.cc:2369] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:08:16.402850 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.400383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.31.115.194:42221)
0620 08:08:16.400474 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:16.400476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:16.400483 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:08:16.400519 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:16.400520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:16.400539 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:16.400598 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:16.400713 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.401181 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.402107 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:16.402302 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:16.402368 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:16.402547 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:16.402653 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:16.402718 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:16.402750 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 08:08:16.402768 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 08:08:16.404819 17701 raft_consensus.cc:1261] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 1 FOLLOWER]: Refusing update from remote peer 9a5fddda74b1409490bca07311c9c0fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:16.404887 17439 raft_consensus.cc:1261] T 674688da098b420a882762b71bb17a90 P 6814a07f8a7640f8818c49636972584c [term 1 FOLLOWER]: Refusing update from remote peer 9a5fddda74b1409490bca07311c9c0fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:16.405257 17771 consensus_queue.cc:1035] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6814a07f8a7640f8818c49636972584c" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:16.405406 17769 consensus_queue.cc:1035] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78de6e45bb646f8b6c0f9ae94f92444" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:16.409312 17776 mvcc.cc:205] Tried to move back new op lower bound from 7298838103671115776 to 7298838103472152576. Current Snapshot: MvccSnapshot[applied={T|T < 7298838103671115776 or (T in {7298838103671115776})}]
I20260620 08:08:16.410683 17296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:16.410795 17296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:16.411685 17296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1274bf11fa01415187cb447acc65fe84: Sending DeleteTablet for 3 replicas of tablet 674688da098b420a882762b71bb17a90
I20260620 08:08:16.412078 17550 tablet_service.cc:1514] Processing DeleteTablet for tablet 674688da098b420a882762b71bb17a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:16 UTC) from {username='slave'} at 127.0.0.1:33678
I20260620 08:08:16.412087 17419 tablet_service.cc:1514] Processing DeleteTablet for tablet 674688da098b420a882762b71bb17a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:16 UTC) from {username='slave'} at 127.0.0.1:34700
I20260620 08:08:16.412278 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17357
I20260620 08:08:16.412442 17784 tablet_replica.cc:330] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: stopping tablet replica
I20260620 08:08:16.412415 17681 tablet_service.cc:1514] Processing DeleteTablet for tablet 674688da098b420a882762b71bb17a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:16 UTC) from {username='slave'} at 127.0.0.1:41748
I20260620 08:08:16.412572 17784 raft_consensus.cc:2229] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:08:16.412639 17785 tablet_replica.cc:330] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: stopping tablet replica
I20260620 08:08:16.412703 17784 raft_consensus.cc:2258] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:16.412747 17785 raft_consensus.cc:2229] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:16.412871 17785 raft_consensus.cc:2258] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:16.413293 17784 ts_tablet_manager.cc:1905] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:16.413293 17785 ts_tablet_manager.cc:1905] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:16.415344 17784 ts_tablet_manager.cc:1918] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:16.415344 17785 ts_tablet_manager.cc:1918] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:16.415455 17785 log.cc:1196] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/674688da098b420a882762b71bb17a90
I20260620 08:08:16.415464 17784 log.cc:1196] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/674688da098b420a882762b71bb17a90
I20260620 08:08:16.415746 17785 ts_tablet_manager.cc:1939] T 674688da098b420a882762b71bb17a90 P f78de6e45bb646f8b6c0f9ae94f92444: Deleting consensus metadata
I20260620 08:08:16.415746 17784 ts_tablet_manager.cc:1939] T 674688da098b420a882762b71bb17a90 P 9a5fddda74b1409490bca07311c9c0fe: Deleting consensus metadata
I20260620 08:08:16.416208 17281 catalog_manager.cc:4841] TS f78de6e45bb646f8b6c0f9ae94f92444 (127.31.115.195:37273): tablet 674688da098b420a882762b71bb17a90 (table test-table [id=25315729152c48cb816a9bc2a8759ff6]) successfully deleted
I20260620 08:08:16.416338 17284 catalog_manager.cc:4841] TS 9a5fddda74b1409490bca07311c9c0fe (127.31.115.194:42221): tablet 674688da098b420a882762b71bb17a90 (table test-table [id=25315729152c48cb816a9bc2a8759ff6]) successfully deleted
W20260620 08:08:16.421707 17281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:43791 (error 108)
I20260620 08:08:16.421784 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17488
I20260620 08:08:16.423439 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:16.422761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:43791 (local address 127.0.0.1:34712)
0620 08:08:16.423320 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:16.423378 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522}
W20260620 08:08:16.423645 17281 catalog_manager.cc:4584] TS 6814a07f8a7640f8818c49636972584c (127.31.115.193:43791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 674688da098b420a882762b71bb17a90: Network error: Client connection negotiation failed: client connection to 127.31.115.193:43791: connect: Connection refused (error 111)
I20260620 08:08:16.427631 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17619
I20260620 08:08:16.432969 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17266
2026-06-20T08:08:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (936 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:16 dist-test-slave-c4w8 krb5kdc[17789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:16 dist-test-slave-c4w8 krb5kdc[17789](info): set up 2 sockets
Jun 20 08:08:16 dist-test-slave-c4w8 krb5kdc[17789](info): commencing operation
krb5kdc: starting...
W20260620 08:08:17.522001 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:17 dist-test-slave-c4w8 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:17.653963 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:32841
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:17.739140 17802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:17.739415 17802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:17.739459 17802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:17.740847 17802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:17.740926 17802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:17.740952 17802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:17.740978 17802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:17.741009 17802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:17.742650 17802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:17.742967 17802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:17.743229 17802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:17.748636 17808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:17.748703 17802 server_base.cc:1034] running on GCE node
W20260620 08:08:17.748636 17807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:17.748641 17810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:17.749161 17802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:17.749517 17802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:17.750773 17802 hybrid_clock.cc:648] HybridClock initialized: now 1781942897750742 us; error 47 us; skew 500 ppm
Jun 20 08:08:17 dist-test-slave-c4w8 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942897, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:17.753731 17802 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:17.754210 17802 webserver.cc:460] Webserver started at http://127.31.115.254:42673/ using document root <none> and password file <none>
I20260620 08:08:17.754516 17802 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:17.754593 17802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:17.754745 17802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:17.755846 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "3e25e50740994e55834b9a609bbcdfa8"
format_stamp: "Formatted at 2026-06-20 08:08:17 on dist-test-slave-c4w8"
server_key: "529b039a7321e86731641c8895c8dd9d"
server_key_iv: "5aa751138d57eba001a47541497f1642"
server_key_version: "encryptionkey@0"
I20260620 08:08:17.756227 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "3e25e50740994e55834b9a609bbcdfa8"
format_stamp: "Formatted at 2026-06-20 08:08:17 on dist-test-slave-c4w8"
server_key: "529b039a7321e86731641c8895c8dd9d"
server_key_iv: "5aa751138d57eba001a47541497f1642"
server_key_version: "encryptionkey@0"
I20260620 08:08:17.757803 17802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:17.758872 17817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:17.759336 17802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:17.759436 17802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "3e25e50740994e55834b9a609bbcdfa8"
format_stamp: "Formatted at 2026-06-20 08:08:17 on dist-test-slave-c4w8"
server_key: "529b039a7321e86731641c8895c8dd9d"
server_key_iv: "5aa751138d57eba001a47541497f1642"
server_key_version: "encryptionkey@0"
I20260620 08:08:17.759567 17802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:17.779265 17802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:17.781973 17802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:17.782160 17802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:17.787626 17802 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:32841
I20260620 08:08:17.787658 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:32841 every 8 connection(s)
I20260620 08:08:17.788100 17802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:17.789111 17870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:17.789533 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17802
I20260620 08:08:17.789690 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:17.789913 32207 external_mini_cluster.cc:1428] Setting key 78b129b0590bc24d1b4e36a2bfe2f7b7
I20260620 08:08:17.792480 17870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8: Bootstrap starting.
I20260620 08:08:17.793113 17870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:17.793416 17870 log.cc:824] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:17.794165 17870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8: No bootstrap required, opened a new log
I20260620 08:08:17.796386 17870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e50740994e55834b9a609bbcdfa8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32841 } }
I20260620 08:08:17.796554 17870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:17.796607 17870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e25e50740994e55834b9a609bbcdfa8, State: Initialized, Role: FOLLOWER
I20260620 08:08:17.796748 17870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e50740994e55834b9a609bbcdfa8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32841 } }
I20260620 08:08:17.796833 17870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:17.796876 17870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:17.796933 17870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:08:17 dist-test-slave-c4w8 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:17.797564 17870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e50740994e55834b9a609bbcdfa8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32841 } }
I20260620 08:08:17.797731 17870 leader_election.cc:302] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e25e50740994e55834b9a609bbcdfa8; no voters:
I20260620 08:08:17.797925 17870 leader_election.cc:288] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:17.798007 17875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:17.798198 17870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:17.798509 17875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e25e50740994e55834b9a609bbcdfa8, State: Running, Role: LEADER
I20260620 08:08:17.798616 17875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e50740994e55834b9a609bbcdfa8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32841 } }
I20260620 08:08:17.799017 17875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e25e50740994e55834b9a609bbcdfa8. Latest consensus state: current_term: 1 leader_uuid: "3e25e50740994e55834b9a609bbcdfa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e50740994e55834b9a609bbcdfa8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32841 } } }
I20260620 08:08:17.799041 17876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e25e50740994e55834b9a609bbcdfa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e50740994e55834b9a609bbcdfa8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 32841 } } }
I20260620 08:08:17.799273 17875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:17.799286 17876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:17.799588 17885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:17.801936 17885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:17.802683 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:17.790554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.31.115.254:32841)
0620 08:08:17.790768 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:17.790776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:17.790804 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:08:17.791268 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:17.791275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:17.791326 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:17.791487 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:17.792313 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:17.793584 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:17.794457 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:17.795060 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:17.795179 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:17.797779 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:17.797784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:17.797784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:17.797798 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:17.800272 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:17.800821 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:17.800825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:17.800826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:17.801019 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:17.801363 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:17.801367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:17.801367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:17.801844 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:17.802032 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:17.802180 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 08:08:17.802356 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260620 08:08:17.804260 17885 catalog_manager.cc:1347] Generated new cluster ID: b38362ac494148fdb54f7fed19e5b1f3
I20260620 08:08:17.804320 17885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:17.812599 17885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:17.813217 17885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:17.815984 17885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8: Generated new TSK 0
I20260620 08:08:17.816182 17885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:08:17.848838 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32841
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:17.971069 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:17.971331 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:17.971375 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:17.972651 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:17.972728 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:17.972777 17896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:08:17.974511 17896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:17.974808 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:17.975061 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:17.981220 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:17.981353 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:17.981565 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:17.983184 17896 server_base.cc:1034] running on GCE node
I20260620 08:08:17.983467 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:17.983777 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:17.985001 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781942897984961 us; error 71 us; skew 500 ppm
Jun 20 08:08:17 dist-test-slave-c4w8 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942897, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:17.988368 17896 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:08:17.989053 17896 webserver.cc:460] Webserver started at http://127.31.115.193:32999/ using document root <none> and password file <none>
I20260620 08:08:17.989472 17896 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:17.989535 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:17.989800 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:17.991616 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "0969772cfa974d1f95039aa394d6b894"
format_stamp: "Formatted at 2026-06-20 08:08:17 on dist-test-slave-c4w8"
server_key: "4f913865e5e6f43b38c2cae14bea46a8"
server_key_iv: "1d97f9425528c951df0bc95c8245a2ab"
server_key_version: "encryptionkey@0"
I20260620 08:08:17.992098 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "0969772cfa974d1f95039aa394d6b894"
format_stamp: "Formatted at 2026-06-20 08:08:17 on dist-test-slave-c4w8"
server_key: "4f913865e5e6f43b38c2cae14bea46a8"
server_key_iv: "1d97f9425528c951df0bc95c8245a2ab"
server_key_version: "encryptionkey@0"
I20260620 08:08:17.994259 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 08:08:17.995491 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:17.995700 17896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:17.995819 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "0969772cfa974d1f95039aa394d6b894"
format_stamp: "Formatted at 2026-06-20 08:08:17 on dist-test-slave-c4w8"
server_key: "4f913865e5e6f43b38c2cae14bea46a8"
server_key_iv: "1d97f9425528c951df0bc95c8245a2ab"
server_key_version: "encryptionkey@0"
I20260620 08:08:17.995910 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:18.004498 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:18.007087 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:18.007213 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:18.007472 17896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:18.007839 17896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:18.007879 17896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.007902 17896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:18.007915 17896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.014109 17896 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:33633
I20260620 08:08:18.014204 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:33633 every 8 connection(s)
I20260620 08:08:18.014576 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:08:18.017689 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 17896
I20260620 08:08:18.017848 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:08:18.018069 32207 external_mini_cluster.cc:1428] Setting key 65bb124fcfccde1112e8e0cb61c06c82
Jun 20 08:08:18 dist-test-slave-c4w8 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942897, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:18.025887 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.015425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:58713 (local address 127.31.115.254:32841)
0620 08:08:18.015529 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.015531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.015686 (+ 155us) server_negotiation.cc:407] Connection header received
0620 08:08:18.016198 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.016199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.016210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.016275 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:18.017270 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.018205 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.019286 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.019634 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.019709 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.022537 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:18.022539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:18.022540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:18.022548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:18.024302 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.024814 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.024816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.024816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.024877 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.025155 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.025156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.025157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.025284 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:18.025361 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.025677 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.025785 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 08:08:18.026149 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.015375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:32841 (local address 127.31.115.193:58713)
0620 08:08:18.015537 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.015582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.016079 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.016331 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.016334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.016933 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:18.017143 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.017147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.018348 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.018350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.019197 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.019200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.019731 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.019733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.019878 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.020510 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:18.020519 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:18.022403 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:18.024461 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:18.024476 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:18.024479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:18.024722 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:18.024962 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:18.024963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:18.024964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:18.025079 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:18.025350 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:18.025358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:18.025518 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.025657 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.025836 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260620 08:08:18.026623 18025 heartbeater.cc:347] Connected to a master server at 127.31.115.254:32841
I20260620 08:08:18.026741 18025 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:08:18.026966 18025 heartbeater.cc:511] Master 127.31.115.254:32841 requested a full tablet report, sending...
I20260620 08:08:18.027550 17834 ts_manager.cc:194] Registered new tserver with Master: 0969772cfa974d1f95039aa394d6b894 (127.31.115.193:33633)
I20260620 08:08:18.028311 17834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:58713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:08:18.045454 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32841
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:18.130213 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:18.130471 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:18.130510 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:18.131669 18030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:18.131742 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:18.131789 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:08:18.133282 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:18.133563 18030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:18.133826 18030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:18.138952 18038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.138959 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.139053 18030 server_base.cc:1034] running on GCE node
W20260620 08:08:18.138974 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.139444 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:18.139750 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:18.140946 18030 hybrid_clock.cc:648] HybridClock initialized: now 1781942898140927 us; error 30 us; skew 500 ppm
Jun 20 08:08:18 dist-test-slave-c4w8 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:18.143754 18030 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:08:18.144321 18030 webserver.cc:460] Webserver started at http://127.31.115.194:39909/ using document root <none> and password file <none>
I20260620 08:08:18.144578 18030 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:18.144655 18030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:18.144841 18030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:18.146099 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "6101ef79cd1549cb9179b63bf2347ae1"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "b1e3fa7c6d9ae25a9127590d7ec38d0a"
server_key_iv: "c17d5b8f8e51c76bfb8941f60627a3b6"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.146492 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "6101ef79cd1549cb9179b63bf2347ae1"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "b1e3fa7c6d9ae25a9127590d7ec38d0a"
server_key_iv: "c17d5b8f8e51c76bfb8941f60627a3b6"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.147996 18030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:18.148890 18045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.149106 18030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:18.149219 18030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "6101ef79cd1549cb9179b63bf2347ae1"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "b1e3fa7c6d9ae25a9127590d7ec38d0a"
server_key_iv: "c17d5b8f8e51c76bfb8941f60627a3b6"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.149298 18030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:18.166420 18030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:18.168990 18030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:18.169214 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:18.169499 18030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:18.169902 18030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:18.169970 18030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.170014 18030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:18.170068 18030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.176811 18030 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:41763
I20260620 08:08:18.176862 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:41763 every 8 connection(s)
I20260620 08:08:18.177234 18030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:08:18.181205 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18030
I20260620 08:08:18.181353 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:08:18.181593 32207 external_mini_cluster.cc:1428] Setting key 9bc9d05647b0c870bb0d732754e9a720
Jun 20 08:08:18 dist-test-slave-c4w8 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:18.191001 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.178026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:32841 (local address 127.31.115.194:59291)
0620 08:08:18.178205 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.178250 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.178783 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.179057 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.179061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.179726 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:18.179937 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.179942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.180720 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.180721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.182180 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.182183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.182734 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.182735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.182925 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.183728 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:18.183741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:18.186041 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:18.188780 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:18.188792 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:18.188794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:18.189095 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:18.189400 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:18.189401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:18.189401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:18.189972 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:18.190400 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:18.190408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:18.190556 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.190654 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.190804 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260620 08:08:18.190997 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.178062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:59291 (local address 127.31.115.254:32841)
0620 08:08:18.178118 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.178120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.178462 (+ 342us) server_negotiation.cc:407] Connection header received
0620 08:08:18.178897 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.178898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.178909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.178977 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:18.180060 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.180588 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.182343 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.182615 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.182681 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.186171 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:18.186173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:18.186173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:18.186184 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:18.188635 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.189193 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.189195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.189196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.189274 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.190066 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.190067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.190067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.190259 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:18.190352 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.190735 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.190872 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:18.191656 18159 heartbeater.cc:347] Connected to a master server at 127.31.115.254:32841
I20260620 08:08:18.191797 18159 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:18.192050 18159 heartbeater.cc:511] Master 127.31.115.254:32841 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
I20260620 08:08:18.192575 17834 ts_manager.cc:194] Registered new tserver with Master: 6101ef79cd1549cb9179b63bf2347ae1 (127.31.115.194:41763)
I20260620 08:08:18.192986 17834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:59291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:08:18.215054 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32841
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:18.308178 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:18.308413 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:18.308440 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:18.309599 18164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:18.309710 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:18.309739 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:08:18.311235 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:18.311450 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:18.311630 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:18.316949 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.317050 18164 server_base.cc:1034] running on GCE node
W20260620 08:08:18.317050 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.317183 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.317420 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:18.317749 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:18.318922 18164 hybrid_clock.cc:648] HybridClock initialized: now 1781942898318904 us; error 33 us; skew 500 ppm
Jun 20 08:08:18 dist-test-slave-c4w8 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:18.321538 18164 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:08:18.322064 18164 webserver.cc:460] Webserver started at http://127.31.115.195:41057/ using document root <none> and password file <none>
I20260620 08:08:18.322305 18164 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:18.322351 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:18.322517 18164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:18.323580 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "db98732643c04fa1ac49748014f1c7d3"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "06eafc2cb663333971afcdb57160430c"
server_key_iv: "d83954b44256df524e31d9de21f14f83"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.323943 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "db98732643c04fa1ac49748014f1c7d3"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "06eafc2cb663333971afcdb57160430c"
server_key_iv: "d83954b44256df524e31d9de21f14f83"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.325497 18164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:18.326640 18179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.326881 18164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:18.326992 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "db98732643c04fa1ac49748014f1c7d3"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "06eafc2cb663333971afcdb57160430c"
server_key_iv: "d83954b44256df524e31d9de21f14f83"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.327165 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:18.343745 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:18.346346 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:18.346518 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:18.346783 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:18.347097 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:18.347153 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.347198 18164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:18.347237 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.353631 18164 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:41443
I20260620 08:08:18.353703 18292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:41443 every 8 connection(s)
I20260620 08:08:18.354012 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:08:18 dist-test-slave-c4w8 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:18.361275 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18164
I20260620 08:08:18.361481 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:08:18.361776 32207 external_mini_cluster.cc:1428] Setting key 2cc0d6069c4919135b85e79f5b4a6926
I20260620 08:08:18.364593 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.354776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:32997 (local address 127.31.115.254:32841)
0620 08:08:18.354855 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.354857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.355103 (+ 246us) server_negotiation.cc:407] Connection header received
0620 08:08:18.355693 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.355694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.355705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.355777 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:18.356732 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.357334 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.358470 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.358675 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.358748 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.361111 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:18.361113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:18.361113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:18.361123 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:18.363131 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.363575 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.363575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.363576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.363642 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.363920 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.363921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.363921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.364053 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:18.364131 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.364417 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.364500 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:08:18.364810 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.354743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:32841 (local address 127.31.115.195:32997)
0620 08:08:18.354978 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.355021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.355592 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.355833 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.355837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.356420 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:18.356620 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.356624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.357444 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.357445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.358350 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.358353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.358797 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.358798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.358929 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.359476 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:18.359485 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:18.360960 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:18.363223 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:18.363233 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:18.363235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:18.363485 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:18.363719 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:18.363720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:18.363720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:18.363849 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:18.364122 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:18.364129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:18.364280 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.364386 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.364557 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":138,"threads_started":1}
I20260620 08:08:18.365339 18293 heartbeater.cc:347] Connected to a master server at 127.31.115.254:32841
I20260620 08:08:18.365476 18293 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:18.365715 18293 heartbeater.cc:511] Master 127.31.115.254:32841 requested a full tablet report, sending...
I20260620 08:08:18.366164 17834 ts_manager.cc:194] Registered new tserver with Master: db98732643c04fa1ac49748014f1c7d3 (127.31.115.195:41443)
I20260620 08:08:18.366564 17834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:32997
I20260620 08:08:18.374362 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:08:18.384032 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.376008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.31.115.254:32841)
0620 08:08:18.376108 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.376111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.376279 (+ 168us) server_negotiation.cc:407] Connection header received
0620 08:08:18.376327 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.376328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.376349 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.376415 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:18.377252 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.377855 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.379012 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.379258 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.379348 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.380966 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:18.380968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:18.380969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:18.380980 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:18.382450 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.382977 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.382978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.382979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.383042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:18.383310 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:18.383311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:18.383311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:18.383461 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:18.383670 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.383819 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.383892 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:08:18.386806 17834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:08:18.388587 17834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:08:18.396155 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.391676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.31.115.194:41763)
0620 08:08:18.391948 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.391951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.392075 (+ 124us) server_negotiation.cc:407] Connection header received
0620 08:08:18.392211 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.392214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.392230 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.392306 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:18.392523 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.393327 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.394784 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.395541 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.395662 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.395743 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.395965 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.396031 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":118,"threads_started":1}
I20260620 08:08:18.396142 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.391547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41763 (local address 127.0.0.1:46712)
0620 08:08:18.391939 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.391984 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.392090 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.392330 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.392334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.392336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:18.392425 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.392427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.393432 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.393433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.394661 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.394664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.395664 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.395666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.395813 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.395839 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.395942 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.396045 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":73,"threads_started":1}
I20260620 08:08:18.396466 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.391772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33633 (local address 127.0.0.1:41082)
0620 08:08:18.392291 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.392301 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.392347 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.392620 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.392621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.392624 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:18.392690 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.392692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.394004 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.394005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.395083 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.395086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.396160 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.396161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.396237 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.396243 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.396292 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.396347 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":98,"threads_started":1}
I20260620 08:08:18.396746 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.392041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.31.115.193:33633)
0620 08:08:18.392465 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.392470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.392480 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:08:18.392520 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.392522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.392538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.392611 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:18.392775 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.393879 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.395177 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.395977 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.396058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.396121 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.396614 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.396662 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":99,"threads_started":1}
I20260620 08:08:18.397495 17959 tablet_service.cc:1467] Processing CreateTablet for tablet 05a52bb823354d42a9915be9199eacdb (DEFAULT_TABLE table=test-table [id=1ec3b4f035484573aaee7b64ac0b7c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:18.397495 18093 tablet_service.cc:1467] Processing CreateTablet for tablet 05a52bb823354d42a9915be9199eacdb (DEFAULT_TABLE table=test-table [id=1ec3b4f035484573aaee7b64ac0b7c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:18.397756 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.391557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:41443 (local address 127.0.0.1:36960)
0620 08:08:18.392018 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.392030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.392104 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.392612 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.392613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.392615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:18.392656 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.392658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.395195 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.395195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.396526 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.396529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.397440 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.397441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.397509 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.397514 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.397633 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.397684 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":88,"threads_started":1}
I20260620 08:08:18.397780 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.391639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.31.115.195:41443)
0620 08:08:18.391862 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.391867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.392407 (+ 540us) server_negotiation.cc:407] Connection header received
0620 08:08:18.392451 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.392453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.392470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.392537 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:18.392720 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.393897 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.396630 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.397318 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.397437 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.397493 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.397659 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.397702 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260620 08:08:18.398108 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a52bb823354d42a9915be9199eacdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:18.398109 18093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a52bb823354d42a9915be9199eacdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:18.398483 18227 tablet_service.cc:1467] Processing CreateTablet for tablet 05a52bb823354d42a9915be9199eacdb (DEFAULT_TABLE table=test-table [id=1ec3b4f035484573aaee7b64ac0b7c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:18.398881 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a52bb823354d42a9915be9199eacdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:18.401705 18310 tablet_bootstrap.cc:492] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Bootstrap starting.
I20260620 08:08:18.402271 18312 tablet_bootstrap.cc:492] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Bootstrap starting.
I20260620 08:08:18.402575 18310 tablet_bootstrap.cc:654] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:18.402878 18312 tablet_bootstrap.cc:654] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:18.403164 18310 log.cc:824] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:18.403466 18312 log.cc:824] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:18.403861 18311 tablet_bootstrap.cc:492] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: Bootstrap starting.
I20260620 08:08:18.404289 18312 tablet_bootstrap.cc:492] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: No bootstrap required, opened a new log
I20260620 08:08:18.404297 18310 tablet_bootstrap.cc:492] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: No bootstrap required, opened a new log
I20260620 08:08:18.404377 18312 ts_tablet_manager.cc:1397] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:08:18.404529 18311 tablet_bootstrap.cc:654] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:18.404541 18310 ts_tablet_manager.cc:1397] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:08:18.405130 18311 log.cc:824] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:18.405856 18311 tablet_bootstrap.cc:492] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: No bootstrap required, opened a new log
I20260620 08:08:18.405951 18311 ts_tablet_manager.cc:1397] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:08:18.406554 18310 raft_consensus.cc:348] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.406554 18312 raft_consensus.cc:348] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.406752 18310 raft_consensus.cc:374] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:18.406752 18312 raft_consensus.cc:374] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:18.406791 18312 raft_consensus.cc:729] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6101ef79cd1549cb9179b63bf2347ae1, State: Initialized, Role: FOLLOWER
I20260620 08:08:18.406791 18310 raft_consensus.cc:729] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db98732643c04fa1ac49748014f1c7d3, State: Initialized, Role: FOLLOWER
I20260620 08:08:18.406950 18312 consensus_queue.cc:260] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.406941 18310 consensus_queue.cc:260] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.407261 18312 ts_tablet_manager.cc:1428] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:08:18.407397 18159 heartbeater.cc:503] Master 127.31.115.254:32841 was elected leader, sending a full tablet report...
I20260620 08:08:18.407511 18293 heartbeater.cc:503] Master 127.31.115.254:32841 was elected leader, sending a full tablet report...
I20260620 08:08:18.407768 18310 ts_tablet_manager.cc:1428] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:08:18.407673 18311 raft_consensus.cc:348] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.407881 18311 raft_consensus.cc:374] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:18.407943 18311 raft_consensus.cc:729] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0969772cfa974d1f95039aa394d6b894, State: Initialized, Role: FOLLOWER
I20260620 08:08:18.408309 18311 consensus_queue.cc:260] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.408654 18311 ts_tablet_manager.cc:1428] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 08:08:18.408792 18025 heartbeater.cc:503] Master 127.31.115.254:32841 was elected leader, sending a full tablet report...
W20260620 08:08:18.427930 18160 tablet.cc:2369] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:08:18.468652 18317 raft_consensus.cc:482] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:18.468791 18317 raft_consensus.cc:504] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.469362 18317 leader_election.cc:288] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6101ef79cd1549cb9179b63bf2347ae1 (127.31.115.194:41763), 0969772cfa974d1f95039aa394d6b894 (127.31.115.193:33633)
I20260620 08:08:18.473158 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.469784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:37461 (local address 127.31.115.193:33633)
0620 08:08:18.469838 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.469840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.470030 (+ 190us) server_negotiation.cc:407] Connection header received
0620 08:08:18.470069 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.470070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.470111 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.470182 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:18.470385 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.470866 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.472392 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.472773 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.472867 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.472879 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.473033 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.473079 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:08:18.473186 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.469567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41763 (local address 127.31.115.195:34517)
0620 08:08:18.469688 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.469707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.469792 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.470140 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.470141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.470144 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:18.470229 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.470232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.470938 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.470939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.472257 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.472260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.472867 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.472868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.472982 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.472985 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.473059 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.473098 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":176}
I20260620 08:08:18.473201 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.469652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:34517 (local address 127.31.115.194:41763)
0620 08:08:18.469714 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.469716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.469776 (+ 60us) server_negotiation.cc:407] Connection header received
0620 08:08:18.469881 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.469882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.469894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.469968 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:18.470324 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.470852 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.472357 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.472777 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.472867 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.472879 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.473085 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.473134 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:08:18.473196 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.469695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33633 (local address 127.31.115.195:37461)
0620 08:08:18.469919 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.469929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.469996 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.470198 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.470199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.470201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:18.470300 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.470302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.470983 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.470984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.472264 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.472267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.472867 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.472868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.472958 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.472962 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.473006 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.473060 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":144,"thread_start_us":134,"threads_started":1}
I20260620 08:08:18.473557 18113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a52bb823354d42a9915be9199eacdb" candidate_uuid: "db98732643c04fa1ac49748014f1c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6101ef79cd1549cb9179b63bf2347ae1" is_pre_election: true
I20260620 08:08:18.473640 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a52bb823354d42a9915be9199eacdb" candidate_uuid: "db98732643c04fa1ac49748014f1c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0969772cfa974d1f95039aa394d6b894" is_pre_election: true
I20260620 08:08:18.473747 18113 raft_consensus.cc:2454] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db98732643c04fa1ac49748014f1c7d3 in term 0.
I20260620 08:08:18.473830 17979 raft_consensus.cc:2454] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db98732643c04fa1ac49748014f1c7d3 in term 0.
I20260620 08:08:18.473992 18180 leader_election.cc:302] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6101ef79cd1549cb9179b63bf2347ae1, db98732643c04fa1ac49748014f1c7d3; no voters:
I20260620 08:08:18.474134 18317 raft_consensus.cc:2781] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:08:18.474196 18317 raft_consensus.cc:482] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:08:18.474222 18317 raft_consensus.cc:3037] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:18.475023 18317 raft_consensus.cc:504] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.475167 18317 leader_election.cc:288] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [CANDIDATE]: Term 1 election: Requested vote from peers 6101ef79cd1549cb9179b63bf2347ae1 (127.31.115.194:41763), 0969772cfa974d1f95039aa394d6b894 (127.31.115.193:33633)
I20260620 08:08:18.475339 18113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a52bb823354d42a9915be9199eacdb" candidate_uuid: "db98732643c04fa1ac49748014f1c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6101ef79cd1549cb9179b63bf2347ae1"
I20260620 08:08:18.475446 18113 raft_consensus.cc:3037] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:18.475389 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a52bb823354d42a9915be9199eacdb" candidate_uuid: "db98732643c04fa1ac49748014f1c7d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0969772cfa974d1f95039aa394d6b894"
I20260620 08:08:18.475672 17979 raft_consensus.cc:3037] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:18.476334 18113 raft_consensus.cc:2454] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db98732643c04fa1ac49748014f1c7d3 in term 1.
I20260620 08:08:18.476531 18180 leader_election.cc:302] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6101ef79cd1549cb9179b63bf2347ae1, db98732643c04fa1ac49748014f1c7d3; no voters:
I20260620 08:08:18.476641 18317 raft_consensus.cc:2781] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:18.476655 17979 raft_consensus.cc:2454] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db98732643c04fa1ac49748014f1c7d3 in term 1.
I20260620 08:08:18.476833 18317 raft_consensus.cc:686] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 1 LEADER]: Becoming Leader. State: Replica: db98732643c04fa1ac49748014f1c7d3, State: Running, Role: LEADER
I20260620 08:08:18.476956 18317 consensus_queue.cc:237] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } }
I20260620 08:08:18.478048 17834 catalog_manager.cc:5495] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db98732643c04fa1ac49748014f1c7d3 (127.31.115.195). New cstate: current_term: 1 leader_uuid: "db98732643c04fa1ac49748014f1c7d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db98732643c04fa1ac49748014f1c7d3" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 41443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 } health_report { overall_health: UNKNOWN } } }
I20260620 08:08:18.495342 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.492785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.31.115.195:41443)
0620 08:08:18.492857 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.492859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.492871 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:08:18.492908 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.492909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.492927 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.493008 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:18.493178 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.493739 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.494518 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.494708 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.494766 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.494909 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:18.494996 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:18.495070 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.495157 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.495202 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:08:18.497421 17979 raft_consensus.cc:1261] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 1 FOLLOWER]: Refusing update from remote peer db98732643c04fa1ac49748014f1c7d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:18.497493 18113 raft_consensus.cc:1261] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 1 FOLLOWER]: Refusing update from remote peer db98732643c04fa1ac49748014f1c7d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:18.497855 18317 consensus_queue.cc:1035] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6101ef79cd1549cb9179b63bf2347ae1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:18.497997 18320 consensus_queue.cc:1035] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0969772cfa974d1f95039aa394d6b894" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:18.501276 18325 mvcc.cc:205] Tried to move back new op lower bound from 7298838112242450432 to 7298838112162299904. Current Snapshot: MvccSnapshot[applied={T|T < 7298838112242450432}]
I20260620 08:08:18.503042 17834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:18.503154 17834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:18.504022 17834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e25e50740994e55834b9a609bbcdfa8: Sending DeleteTablet for 3 replicas of tablet 05a52bb823354d42a9915be9199eacdb
I20260620 08:08:18.504416 18093 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a52bb823354d42a9915be9199eacdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:18 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:46712
I20260620 08:08:18.504421 17959 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a52bb823354d42a9915be9199eacdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:18 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:41082
I20260620 08:08:18.504657 18227 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a52bb823354d42a9915be9199eacdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:18 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:36960
I20260620 08:08:18.504678 18333 tablet_replica.cc:330] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: stopping tablet replica
I20260620 08:08:18.504823 18333 raft_consensus.cc:2229] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:18.504976 18333 raft_consensus.cc:2258] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:18.505124 18335 tablet_replica.cc:330] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: stopping tablet replica
I20260620 08:08:18.505170 18334 tablet_replica.cc:330] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894: stopping tablet replica
I20260620 08:08:18.505240 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17896
I20260620 08:08:18.505235 18335 raft_consensus.cc:2229] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:08:18.505272 18334 raft_consensus.cc:2229] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:18.505373 18334 raft_consensus.cc:2258] T 05a52bb823354d42a9915be9199eacdb P 0969772cfa974d1f95039aa394d6b894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:18.505378 18335 raft_consensus.cc:2258] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:18.505945 18333 ts_tablet_manager.cc:1905] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:18.506608 18335 ts_tablet_manager.cc:1905] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:18.507843 18333 ts_tablet_manager.cc:1918] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:18.507915 18333 log.cc:1196] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/05a52bb823354d42a9915be9199eacdb
I20260620 08:08:18.508219 18333 ts_tablet_manager.cc:1939] T 05a52bb823354d42a9915be9199eacdb P 6101ef79cd1549cb9179b63bf2347ae1: Deleting consensus metadata
I20260620 08:08:18.508229 18335 ts_tablet_manager.cc:1918] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:18.508302 18335 log.cc:1196] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/05a52bb823354d42a9915be9199eacdb
I20260620 08:08:18.508571 18335 ts_tablet_manager.cc:1939] T 05a52bb823354d42a9915be9199eacdb P db98732643c04fa1ac49748014f1c7d3: Deleting consensus metadata
I20260620 08:08:18.508651 17818 catalog_manager.cc:4841] TS 6101ef79cd1549cb9179b63bf2347ae1 (127.31.115.194:41763): tablet 05a52bb823354d42a9915be9199eacdb (table test-table [id=1ec3b4f035484573aaee7b64ac0b7c3c]) successfully deleted
I20260620 08:08:18.509078 17820 catalog_manager.cc:4841] TS db98732643c04fa1ac49748014f1c7d3 (127.31.115.195:41443): tablet 05a52bb823354d42a9915be9199eacdb (table test-table [id=1ec3b4f035484573aaee7b64ac0b7c3c]) successfully deleted
W20260620 08:08:18.511983 17818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:33633 (error 108)
I20260620 08:08:18.512373 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18030
I20260620 08:08:18.512511 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.512363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33633 (local address 127.0.0.1:41092)
0620 08:08:18.512426 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.512469 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:08:18.517995 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18164
W20260620 08:08:18.517951 17818 catalog_manager.cc:4584] TS 0969772cfa974d1f95039aa394d6b894 (127.31.115.193:33633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05a52bb823354d42a9915be9199eacdb: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33633: connect: Connection refused (error 111)
I20260620 08:08:18.523419 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 17802
2026-06-20T08:08:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2091 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T08:08:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:18Z Disabled control of system clock
I20260620 08:08:18.552857 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36235
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:18.638204 18340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:18.638443 18340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:18.638466 18340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:18.639642 18340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:18.639693 18340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:18.639704 18340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:18.639717 18340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:18.639729 18340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:18.641068 18340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:36235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:36235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:18.641274 18340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:18.641522 18340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:18.646880 18348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.646907 18346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.646854 18345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.646935 18340 server_base.cc:1034] running on GCE node
I20260620 08:08:18.647441 18340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:18.647758 18340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:18.648952 18340 hybrid_clock.cc:648] HybridClock initialized: now 1781942898648925 us; error 41 us; skew 500 ppm
I20260620 08:08:18.650360 18340 webserver.cc:460] Webserver started at http://127.31.115.254:44999/ using document root <none> and password file <none>
I20260620 08:08:18.650677 18340 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:18.650754 18340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:18.650926 18340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:18.652057 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "bf25417f8f72437ebf3c0a372b95deff"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "f15f49ec708e149cabb8e5ac7b7eab99"
server_key_iv: "f041505aa94baa23ea6a635b9c94ab6b"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.652448 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "bf25417f8f72437ebf3c0a372b95deff"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "f15f49ec708e149cabb8e5ac7b7eab99"
server_key_iv: "f041505aa94baa23ea6a635b9c94ab6b"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.653934 18340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:08:18.654990 18354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.655233 18340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:18.655397 18340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "bf25417f8f72437ebf3c0a372b95deff"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "f15f49ec708e149cabb8e5ac7b7eab99"
server_key_iv: "f041505aa94baa23ea6a635b9c94ab6b"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.655486 18340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:18.677119 18340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:18.677497 18340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:18.677737 18340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:18.682389 18340 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:36235
I20260620 08:08:18.682459 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:36235 every 8 connection(s)
I20260620 08:08:18.682859 18340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:18.684363 18407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:18.687152 18407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff: Bootstrap starting.
I20260620 08:08:18.687717 18407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:18.688059 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18340
I20260620 08:08:18.688062 18407 log.cc:824] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:18.688169 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:18.688374 32207 external_mini_cluster.cc:1428] Setting key db7563c65aa43eb68192cf86515481b3
I20260620 08:08:18.689091 18407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff: No bootstrap required, opened a new log
I20260620 08:08:18.691847 18407 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25417f8f72437ebf3c0a372b95deff" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36235 } }
I20260620 08:08:18.692022 18407 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:18.692052 18407 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25417f8f72437ebf3c0a372b95deff, State: Initialized, Role: FOLLOWER
I20260620 08:08:18.692204 18407 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25417f8f72437ebf3c0a372b95deff" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36235 } }
I20260620 08:08:18.692310 18407 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:18.692358 18407 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:18.692410 18407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:18.693073 18407 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25417f8f72437ebf3c0a372b95deff" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36235 } }
I20260620 08:08:18.693192 18407 leader_election.cc:302] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25417f8f72437ebf3c0a372b95deff; no voters:
I20260620 08:08:18.693372 18407 leader_election.cc:288] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:18.693549 18413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:18.693709 18407 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:18.693728 18413 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [term 1 LEADER]: Becoming Leader. State: Replica: bf25417f8f72437ebf3c0a372b95deff, State: Running, Role: LEADER
I20260620 08:08:18.693871 18413 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25417f8f72437ebf3c0a372b95deff" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36235 } }
I20260620 08:08:18.694211 18413 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf25417f8f72437ebf3c0a372b95deff. Latest consensus state: current_term: 1 leader_uuid: "bf25417f8f72437ebf3c0a372b95deff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25417f8f72437ebf3c0a372b95deff" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36235 } } }
I20260620 08:08:18.694278 18413 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:18.694626 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.689004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.31.115.254:36235)
0620 08:08:18.689288 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.689296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.689323 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:08:18.689829 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.689844 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.689914 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.690029 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:18.690267 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.691155 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.692268 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.692838 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.692934 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.693806 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:18.693811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:18.693811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:18.694048 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:18.694091 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:18.694106 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:18.694174 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.694306 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.694394 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260620 08:08:18.694666 18412 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf25417f8f72437ebf3c0a372b95deff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25417f8f72437ebf3c0a372b95deff" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 36235 } } }
I20260620 08:08:18.694737 18412 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:18.694772 18418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:18.695453 18418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:18.697192 18418 catalog_manager.cc:1347] Generated new cluster ID: f0c85194639542fd8ea7e6e76c3225b3
I20260620 08:08:18.697252 18418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:18.721805 18418 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:18.722388 18418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:18.727622 18418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff: Generated new TSK 0
I20260620 08:08:18.727856 18418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:18.735150 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36235
--builtin_ntp_servers=127.31.115.212:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:18.824590 18431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:18.824836 18431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:18.824862 18431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:18.826272 18431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:18.826325 18431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:18.826359 18431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:08:18.827939 18431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:18.828248 18431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:18.828613 18431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:18.835189 18436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.835176 18437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.835427 18431 server_base.cc:1034] running on GCE node
W20260620 08:08:18.835248 18439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.835732 18431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:18.836015 18431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:18.837194 18431 hybrid_clock.cc:648] HybridClock initialized: now 1781942898837180 us; error 31 us; skew 500 ppm
I20260620 08:08:18.838613 18431 webserver.cc:460] Webserver started at http://127.31.115.193:33141/ using document root <none> and password file <none>
I20260620 08:08:18.838876 18431 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:18.838958 18431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:18.839126 18431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:18.840238 18431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "fdeea12291b644eba373ec9b825b2faa"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "397fcc74f4411c88906274658c4ec7b5"
server_key_iv: "209c54e00302860416f5846276a10fed"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.840817 18431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "fdeea12291b644eba373ec9b825b2faa"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "397fcc74f4411c88906274658c4ec7b5"
server_key_iv: "209c54e00302860416f5846276a10fed"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.842458 18431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 08:08:18.843497 18445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.843766 18431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:18.843883 18431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "fdeea12291b644eba373ec9b825b2faa"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "397fcc74f4411c88906274658c4ec7b5"
server_key_iv: "209c54e00302860416f5846276a10fed"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.843967 18431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:18.857991 18431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:18.858337 18431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:18.858494 18431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:18.858758 18431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:18.859091 18431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:18.859145 18431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.859190 18431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:18.859226 18431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.866114 18431 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:41637
I20260620 08:08:18.866163 18558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:41637 every 8 connection(s)
I20260620 08:08:18.866544 18431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:08:18.871106 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18431
I20260620 08:08:18.871240 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:08:18.871455 32207 external_mini_cluster.cc:1428] Setting key 1355e65ede6b36a2ba485e4fa664ed9f
I20260620 08:08:18.872812 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.867449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:33681 (local address 127.31.115.254:36235)
0620 08:08:18.867543 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:18.867545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:18.867794 (+ 249us) server_negotiation.cc:407] Connection header received
0620 08:08:18.868307 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:18.868308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:18.868326 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:18.868423 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:18.869145 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.869864 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.870938 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.871250 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.871338 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.872459 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:18.872461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:18.872461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:18.872467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:18.872487 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:18.872493 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:18.872555 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:18.872660 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 08:08:18.872692 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 08:08:18.872854 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36235
--builtin_ntp_servers=127.31.115.212:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:08:18.872957 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:18.867430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36235 (local address 127.31.115.193:33681)
0620 08:08:18.867623 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:18.867674 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:18.868172 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:18.868493 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:18.868497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:18.868509 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:18.869029 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.869033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.869994 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.869995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.870825 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:18.870828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:18.871400 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:18.871402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:18.871567 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:18.872272 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:18.872273 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:18.872316 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:18.872347 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:18.872571 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:18.872577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:18.872580 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:18.872640 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:08:18.872727 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 08:08:18.873582 18559 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36235
I20260620 08:08:18.873776 18559 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:18.873984 18559 heartbeater.cc:511] Master 127.31.115.254:36235 requested a full tablet report, sending...
I20260620 08:08:18.874640 18370 ts_manager.cc:194] Registered new tserver with Master: fdeea12291b644eba373ec9b825b2faa (127.31.115.193:41637)
I20260620 08:08:18.875276 18370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.193:33681
W20260620 08:08:18.964828 18562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:18.965065 18562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:18.965090 18562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:18.966426 18562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:18.966483 18562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:18.966513 18562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:08:18.967952 18562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:18.968307 18562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:18.968657 18562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:18.974418 18568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.974992 18567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:18.975034 18570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:18.975092 18562 server_base.cc:1034] running on GCE node
I20260620 08:08:18.975392 18562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:18.975620 18562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:18.976774 18562 hybrid_clock.cc:648] HybridClock initialized: now 1781942898976755 us; error 41 us; skew 500 ppm
I20260620 08:08:18.978139 18562 webserver.cc:460] Webserver started at http://127.31.115.194:35257/ using document root <none> and password file <none>
I20260620 08:08:18.978454 18562 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:18.978505 18562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:18.978641 18562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:18.979748 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "5fe5e05e8b844e2bb285c23d394d32d1"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "0791d712b8589eea9cecabe1d6d6ab74"
server_key_iv: "82882864859f123a2a7984eafee49f3f"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.980123 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "5fe5e05e8b844e2bb285c23d394d32d1"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "0791d712b8589eea9cecabe1d6d6ab74"
server_key_iv: "82882864859f123a2a7984eafee49f3f"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.981674 18562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:08:18.982673 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.982970 18562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:18.983053 18562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "5fe5e05e8b844e2bb285c23d394d32d1"
format_stamp: "Formatted at 2026-06-20 08:08:18 on dist-test-slave-c4w8"
server_key: "0791d712b8589eea9cecabe1d6d6ab74"
server_key_iv: "82882864859f123a2a7984eafee49f3f"
server_key_version: "encryptionkey@0"
I20260620 08:08:18.983153 18562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:18.993544 18562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:18.993995 18562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:18.994187 18562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:18.994446 18562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:18.994792 18562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:18.994850 18562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:18.994890 18562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:18.994958 18562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:19.001526 18562 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:40967
I20260620 08:08:19.001569 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:40967 every 8 connection(s)
I20260620 08:08:19.002010 18562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:08:19.008186 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.002981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:53507 (local address 127.31.115.254:36235)
0620 08:08:19.003068 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.003070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.003385 (+ 315us) server_negotiation.cc:407] Connection header received
0620 08:08:19.003916 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.003917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.003928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.003996 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:19.004674 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.005269 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.006444 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.006654 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.006832 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.007889 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:19.007890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:19.007890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:19.007896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:19.007911 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:19.007915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:19.007969 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.008085 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.008113 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:08:19.008361 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.003036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36235 (local address 127.31.115.194:53507)
0620 08:08:19.003237 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.003273 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.003764 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.004050 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.004053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.004061 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:19.004571 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.004575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.005434 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.005435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.006320 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.006322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.006874 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.006876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.007001 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.007705 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:19.007705 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:19.007737 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:19.007763 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:19.007980 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:19.007985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:19.007987 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.008042 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.008129 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260620 08:08:19.009013 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18562
I20260620 08:08:19.009084 18690 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36235
I20260620 08:08:19.009164 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:08:19.009187 18690 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:19.009420 18690 heartbeater.cc:511] Master 127.31.115.254:36235 requested a full tablet report, sending...
I20260620 08:08:19.009447 32207 external_mini_cluster.cc:1428] Setting key 2dbbfd389272b4c0b6c681cbfcfc815e
I20260620 08:08:19.009996 18371 ts_manager.cc:194] Registered new tserver with Master: 5fe5e05e8b844e2bb285c23d394d32d1 (127.31.115.194:40967)
I20260620 08:08:19.010553 18371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.194:53507
I20260620 08:08:19.010596 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36235
--builtin_ntp_servers=127.31.115.212:37911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:19.097143 18693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:19.097371 18693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:19.097396 18693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:19.098773 18693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:19.098821 18693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:19.098850 18693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:08:19.100306 18693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:37911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:19.100580 18693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:19.100932 18693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:19.106213 18701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:19.106397 18693 server_base.cc:1034] running on GCE node
W20260620 08:08:19.106226 18698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:19.106235 18699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:19.106698 18693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:19.107003 18693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:19.108155 18693 hybrid_clock.cc:648] HybridClock initialized: now 1781942899108136 us; error 38 us; skew 500 ppm
I20260620 08:08:19.109454 18693 webserver.cc:460] Webserver started at http://127.31.115.195:39525/ using document root <none> and password file <none>
I20260620 08:08:19.109797 18693 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:19.109879 18693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:19.110046 18693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:19.111148 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "ef5d5e1f22084b768c7592471874a33f"
format_stamp: "Formatted at 2026-06-20 08:08:19 on dist-test-slave-c4w8"
server_key: "2ed7e45367ece1cd47732384a83d495f"
server_key_iv: "dfd977b38e31d9395e4fb6b35112fa7a"
server_key_version: "encryptionkey@0"
I20260620 08:08:19.111534 18693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "ef5d5e1f22084b768c7592471874a33f"
format_stamp: "Formatted at 2026-06-20 08:08:19 on dist-test-slave-c4w8"
server_key: "2ed7e45367ece1cd47732384a83d495f"
server_key_iv: "dfd977b38e31d9395e4fb6b35112fa7a"
server_key_version: "encryptionkey@0"
I20260620 08:08:19.112911 18693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:19.113842 18707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:19.114042 18693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:19.114156 18693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "ef5d5e1f22084b768c7592471874a33f"
format_stamp: "Formatted at 2026-06-20 08:08:19 on dist-test-slave-c4w8"
server_key: "2ed7e45367ece1cd47732384a83d495f"
server_key_iv: "dfd977b38e31d9395e4fb6b35112fa7a"
server_key_version: "encryptionkey@0"
I20260620 08:08:19.114235 18693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:19.123623 18693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:19.123991 18693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:19.124154 18693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:19.124404 18693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:19.124763 18693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:19.124819 18693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:19.124861 18693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:19.124889 18693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:19.130901 18693 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:37339
I20260620 08:08:19.130954 18820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:37339 every 8 connection(s)
I20260620 08:08:19.131292 18693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:08:19.136368 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18693
I20260620 08:08:19.136502 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:08:19.136747 32207 external_mini_cluster.cc:1428] Setting key 04fdce794dc6cbe76d5909ae82176375
I20260620 08:08:19.136893 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.132035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:37551 (local address 127.31.115.254:36235)
0620 08:08:19.132083 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.132085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.132365 (+ 280us) server_negotiation.cc:407] Connection header received
0620 08:08:19.132834 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.132835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.132846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.132911 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:19.133581 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.134171 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.135203 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.135435 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.135502 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.136619 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:19.136621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:19.136622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:19.136628 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:19.136648 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:19.136653 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:19.136712 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.136779 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.136810 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:08:19.137166 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.132000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:36235 (local address 127.31.115.195:37551)
0620 08:08:19.132193 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.132234 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.132717 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.132959 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.132963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.132971 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:19.133479 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.133483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.134280 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.134281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.135109 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.135112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.135556 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.135558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.135736 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.136419 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:19.136420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:19.136451 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:19.136528 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:19.136729 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:19.136734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:19.136736 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.136779 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.136929 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260620 08:08:19.137907 18821 heartbeater.cc:347] Connected to a master server at 127.31.115.254:36235
I20260620 08:08:19.138012 18821 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:19.138197 18821 heartbeater.cc:511] Master 127.31.115.254:36235 requested a full tablet report, sending...
I20260620 08:08:19.138592 18371 ts_manager.cc:194] Registered new tserver with Master: ef5d5e1f22084b768c7592471874a33f (127.31.115.195:37339)
I20260620 08:08:19.138984 18371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.195:37551
I20260620 08:08:19.148909 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:08:19.154109 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.150652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.31.115.254:36235)
0620 08:08:19.150723 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.150726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.150840 (+ 114us) server_negotiation.cc:407] Connection header received
0620 08:08:19.150903 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.150904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.150914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.150963 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:19.151122 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.151636 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.152533 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.152725 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.152789 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.153708 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:19.153710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:19.153710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:19.153715 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:19.153730 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:19.153734 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:19.153834 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.153955 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.153989 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:08:19.156112 18371 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:08:19.158010 18371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:08:19.165369 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.161197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.31.115.193:41637)
0620 08:08:19.161675 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.161679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.161692 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:08:19.161724 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.161725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.161755 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.161840 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:19.162135 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.162880 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.164165 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.164917 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.165005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.165128 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.165220 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.165265 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":89,"threads_started":1}
I20260620 08:08:19.165376 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.160808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:41637 (local address 127.0.0.1:40090)
0620 08:08:19.161225 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.161234 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.161272 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.162017 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.162018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.162019 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:19.162056 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.162058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.162986 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.162987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.164079 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.164082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.165022 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.165023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.165108 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.165124 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.165204 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.165265 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":78,"threads_started":1}
I20260620 08:08:19.165570 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.160972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.31.115.195:37339)
0620 08:08:19.161165 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.161171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.161180 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:08:19.161218 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.161220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.161234 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.161333 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:19.161549 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.162709 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.164291 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.164982 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.165045 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.165128 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.165418 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.165463 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260620 08:08:19.165583 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.161006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.31.115.194:40967)
0620 08:08:19.161409 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.161412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.161420 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:08:19.161452 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.161453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.161464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.161947 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:19.162123 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.163217 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.164459 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.165030 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.165106 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.165205 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.165438 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.165486 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":92,"threads_started":1}
I20260620 08:08:19.165645 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.160791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:37339 (local address 127.0.0.1:60488)
0620 08:08:19.160993 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.161039 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.161106 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.161382 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.161386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.161388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:19.161469 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.161470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.162805 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.162806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.164183 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.164186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.165167 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.165168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.165237 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.165240 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.165291 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.165335 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":30,"thread_start_us":79,"threads_started":1}
I20260620 08:08:19.165848 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.160795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:40967 (local address 127.0.0.1:50722)
0620 08:08:19.161082 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.161093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.161145 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.161975 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.161976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.161978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:19.162026 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.162027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.163306 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.163307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.164377 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.164379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.165241 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.165242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.165292 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.165306 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.165354 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.165394 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":87,"threads_started":1}
I20260620 08:08:19.166435 18755 tablet_service.cc:1467] Processing CreateTablet for tablet dd89830cf3a24f179a73fb2db057ac4e (DEFAULT_TABLE table=test-table [id=911c8aa7053d4e42b5c18e1c1e945418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:19.166568 18624 tablet_service.cc:1467] Processing CreateTablet for tablet dd89830cf3a24f179a73fb2db057ac4e (DEFAULT_TABLE table=test-table [id=911c8aa7053d4e42b5c18e1c1e945418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:19.166877 18755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd89830cf3a24f179a73fb2db057ac4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:19.166916 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd89830cf3a24f179a73fb2db057ac4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:19.169641 18835 tablet_bootstrap.cc:492] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Bootstrap starting.
I20260620 08:08:19.169977 18836 tablet_bootstrap.cc:492] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Bootstrap starting.
I20260620 08:08:19.170199 18493 tablet_service.cc:1467] Processing CreateTablet for tablet dd89830cf3a24f179a73fb2db057ac4e (DEFAULT_TABLE table=test-table [id=911c8aa7053d4e42b5c18e1c1e945418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:19.170414 18835 tablet_bootstrap.cc:654] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:19.170584 18493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd89830cf3a24f179a73fb2db057ac4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:19.170655 18836 tablet_bootstrap.cc:654] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:19.170748 18835 log.cc:824] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:19.170955 18836 log.cc:824] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:19.171643 18835 tablet_bootstrap.cc:492] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: No bootstrap required, opened a new log
I20260620 08:08:19.171670 18836 tablet_bootstrap.cc:492] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: No bootstrap required, opened a new log
I20260620 08:08:19.171718 18835 ts_tablet_manager.cc:1397] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:08:19.171751 18836 ts_tablet_manager.cc:1397] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:08:19.173542 18836 raft_consensus.cc:348] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.173758 18836 raft_consensus.cc:374] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:19.173817 18836 raft_consensus.cc:729] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5d5e1f22084b768c7592471874a33f, State: Initialized, Role: FOLLOWER
I20260620 08:08:19.173951 18836 consensus_queue.cc:260] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.174243 18836 ts_tablet_manager.cc:1428] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:08:19.174368 18821 heartbeater.cc:503] Master 127.31.115.254:36235 was elected leader, sending a full tablet report...
I20260620 08:08:19.174592 18840 tablet_bootstrap.cc:492] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: Bootstrap starting.
I20260620 08:08:19.174500 18835 raft_consensus.cc:348] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.174672 18835 raft_consensus.cc:374] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:19.174713 18835 raft_consensus.cc:729] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe5e05e8b844e2bb285c23d394d32d1, State: Initialized, Role: FOLLOWER
I20260620 08:08:19.174861 18835 consensus_queue.cc:260] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.175143 18835 ts_tablet_manager.cc:1428] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260620 08:08:19.175206 18690 heartbeater.cc:503] Master 127.31.115.254:36235 was elected leader, sending a full tablet report...
I20260620 08:08:19.175313 18840 tablet_bootstrap.cc:654] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:19.175618 18840 log.cc:824] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:19.176472 18840 tablet_bootstrap.cc:492] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: No bootstrap required, opened a new log
I20260620 08:08:19.176554 18840 ts_tablet_manager.cc:1397] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:08:19.178555 18840 raft_consensus.cc:348] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.178772 18840 raft_consensus.cc:374] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:19.178813 18840 raft_consensus.cc:729] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdeea12291b644eba373ec9b825b2faa, State: Initialized, Role: FOLLOWER
I20260620 08:08:19.178942 18840 consensus_queue.cc:260] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.179252 18840 ts_tablet_manager.cc:1428] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:08:19.179343 18559 heartbeater.cc:503] Master 127.31.115.254:36235 was elected leader, sending a full tablet report...
I20260620 08:08:19.218327 18843 raft_consensus.cc:482] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:19.218487 18843 raft_consensus.cc:504] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.218930 18843 leader_election.cc:288] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5d5e1f22084b768c7592471874a33f (127.31.115.195:37339), 5fe5e05e8b844e2bb285c23d394d32d1 (127.31.115.194:40967)
I20260620 08:08:19.222663 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.219198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:36683 (local address 127.31.115.194:40967)
0620 08:08:19.219255 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.219258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.219267 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:08:19.219322 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.219323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.219333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.219386 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:19.219600 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.220328 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.221928 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.222298 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.222408 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.222418 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.222544 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.222593 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:08:19.222659 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.219118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:40967 (local address 127.31.115.193:36683)
0620 08:08:19.219186 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.219197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.219243 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.219446 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.219448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.219451 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:19.219531 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.219533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.220458 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.220459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.221810 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.221814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.222385 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.222387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.222465 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.222468 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.222524 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.222559 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":35}
I20260620 08:08:19.222735 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.219468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:58133 (local address 127.31.115.195:37339)
0620 08:08:19.219535 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.219537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.219566 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:08:19.219641 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.219642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.219651 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.219706 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:19.219938 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.220564 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.221936 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.222304 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.222403 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.222414 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.222616 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.222662 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:08:19.222770 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.219119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:37339 (local address 127.31.115.193:58133)
0620 08:08:19.219495 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.219507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:19.219583 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:19.219782 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:19.219784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:19.219786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:19.219850 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.219851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.220712 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.220713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.221847 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:19.221849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.222425 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.222427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:19.222531 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.222536 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:08:19.222596 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:08:19.222635 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":171,"thread_start_us":121,"threads_started":1}
I20260620 08:08:19.223109 18644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd89830cf3a24f179a73fb2db057ac4e" candidate_uuid: "fdeea12291b644eba373ec9b825b2faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" is_pre_election: true
I20260620 08:08:19.223188 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd89830cf3a24f179a73fb2db057ac4e" candidate_uuid: "fdeea12291b644eba373ec9b825b2faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5d5e1f22084b768c7592471874a33f" is_pre_election: true
I20260620 08:08:19.223291 18644 raft_consensus.cc:2454] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdeea12291b644eba373ec9b825b2faa in term 0.
I20260620 08:08:19.223310 18775 raft_consensus.cc:2454] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdeea12291b644eba373ec9b825b2faa in term 0.
I20260620 08:08:19.223541 18446 leader_election.cc:302] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fe5e05e8b844e2bb285c23d394d32d1, fdeea12291b644eba373ec9b825b2faa; no voters:
I20260620 08:08:19.223680 18843 raft_consensus.cc:2781] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:08:19.223740 18843 raft_consensus.cc:482] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:08:19.223776 18843 raft_consensus.cc:3037] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:19.224540 18843 raft_consensus.cc:504] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.224701 18843 leader_election.cc:288] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [CANDIDATE]: Term 1 election: Requested vote from peers ef5d5e1f22084b768c7592471874a33f (127.31.115.195:37339), 5fe5e05e8b844e2bb285c23d394d32d1 (127.31.115.194:40967)
I20260620 08:08:19.224898 18644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd89830cf3a24f179a73fb2db057ac4e" candidate_uuid: "fdeea12291b644eba373ec9b825b2faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe5e05e8b844e2bb285c23d394d32d1"
I20260620 08:08:19.224911 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd89830cf3a24f179a73fb2db057ac4e" candidate_uuid: "fdeea12291b644eba373ec9b825b2faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5d5e1f22084b768c7592471874a33f"
I20260620 08:08:19.224992 18644 raft_consensus.cc:3037] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:19.224992 18775 raft_consensus.cc:3037] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:19.225975 18775 raft_consensus.cc:2454] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdeea12291b644eba373ec9b825b2faa in term 1.
I20260620 08:08:19.225966 18644 raft_consensus.cc:2454] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdeea12291b644eba373ec9b825b2faa in term 1.
I20260620 08:08:19.226223 18446 leader_election.cc:302] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fe5e05e8b844e2bb285c23d394d32d1, fdeea12291b644eba373ec9b825b2faa; no voters:
I20260620 08:08:19.226369 18843 raft_consensus.cc:2781] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:19.226575 18843 raft_consensus.cc:686] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 1 LEADER]: Becoming Leader. State: Replica: fdeea12291b644eba373ec9b825b2faa, State: Running, Role: LEADER
I20260620 08:08:19.226686 18843 consensus_queue.cc:237] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } }
I20260620 08:08:19.227644 18370 catalog_manager.cc:5495] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa reported cstate change: term changed from 0 to 1, leader changed from <none> to fdeea12291b644eba373ec9b825b2faa (127.31.115.193). New cstate: current_term: 1 leader_uuid: "fdeea12291b644eba373ec9b825b2faa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdeea12291b644eba373ec9b825b2faa" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 41637 } health_report { overall_health: HEALTHY } } }
I20260620 08:08:19.242316 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.239526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.31.115.193:41637)
0620 08:08:19.239590 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:19.239592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:19.239600 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:08:19.239637 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:19.239638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:19.239657 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:19.239718 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:19.239889 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.240474 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.241291 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:19.241552 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:19.241696 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:19.241864 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:19.241959 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:19.242039 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:19.242143 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 08:08:19.242176 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:08:19.244549 18644 raft_consensus.cc:1261] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 1 FOLLOWER]: Refusing update from remote peer fdeea12291b644eba373ec9b825b2faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:19.244575 18775 raft_consensus.cc:1261] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 1 FOLLOWER]: Refusing update from remote peer fdeea12291b644eba373ec9b825b2faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:19.244971 18845 consensus_queue.cc:1035] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5d5e1f22084b768c7592471874a33f" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 37339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:19.245121 18843 consensus_queue.cc:1035] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fe5e05e8b844e2bb285c23d394d32d1" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:19.247750 18848 mvcc.cc:205] Tried to move back new op lower bound from 7298838115302273024 to 7298838115233153024. Current Snapshot: MvccSnapshot[applied={T|T < 7298838115302273024}]
I20260620 08:08:19.248795 18849 mvcc.cc:205] Tried to move back new op lower bound from 7298838115302273024 to 7298838115233153024. Current Snapshot: MvccSnapshot[applied={T|T < 7298838115302273024 or (T in {7298838115302273024})}]
I20260620 08:08:19.250362 18370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:19.250489 18370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:19.251312 18370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf25417f8f72437ebf3c0a372b95deff: Sending DeleteTablet for 3 replicas of tablet dd89830cf3a24f179a73fb2db057ac4e
I20260620 08:08:19.251653 18493 tablet_service.cc:1514] Processing DeleteTablet for tablet dd89830cf3a24f179a73fb2db057ac4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:19 UTC) from {username='slave'} at 127.0.0.1:40090
I20260620 08:08:19.251664 18755 tablet_service.cc:1514] Processing DeleteTablet for tablet dd89830cf3a24f179a73fb2db057ac4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:19 UTC) from {username='slave'} at 127.0.0.1:60488
I20260620 08:08:19.251739 18624 tablet_service.cc:1514] Processing DeleteTablet for tablet dd89830cf3a24f179a73fb2db057ac4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:19 UTC) from {username='slave'} at 127.0.0.1:50722
I20260620 08:08:19.251909 18859 tablet_replica.cc:330] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa: stopping tablet replica
I20260620 08:08:19.252017 18859 raft_consensus.cc:2229] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:08:19.252097 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18431
I20260620 08:08:19.252120 18859 raft_consensus.cc:2258] T dd89830cf3a24f179a73fb2db057ac4e P fdeea12291b644eba373ec9b825b2faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:19.253123 18858 tablet_replica.cc:330] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: stopping tablet replica
I20260620 08:08:19.253239 18858 raft_consensus.cc:2229] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:19.253360 18858 raft_consensus.cc:2258] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:19.253540 18860 tablet_replica.cc:330] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: stopping tablet replica
I20260620 08:08:19.253669 18860 raft_consensus.cc:2229] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:19.253775 18860 raft_consensus.cc:2258] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:19.254158 18858 ts_tablet_manager.cc:1905] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:19.254212 18860 ts_tablet_manager.cc:1905] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:19.255927 18858 ts_tablet_manager.cc:1918] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:19.256036 18858 log.cc:1196] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/dd89830cf3a24f179a73fb2db057ac4e
I20260620 08:08:19.256175 18860 ts_tablet_manager.cc:1918] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:19.256258 18860 log.cc:1196] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/dd89830cf3a24f179a73fb2db057ac4e
I20260620 08:08:19.256377 18858 ts_tablet_manager.cc:1939] T dd89830cf3a24f179a73fb2db057ac4e P ef5d5e1f22084b768c7592471874a33f: Deleting consensus metadata
I20260620 08:08:19.256513 18860 ts_tablet_manager.cc:1939] T dd89830cf3a24f179a73fb2db057ac4e P 5fe5e05e8b844e2bb285c23d394d32d1: Deleting consensus metadata
I20260620 08:08:19.256956 18357 catalog_manager.cc:4841] TS ef5d5e1f22084b768c7592471874a33f (127.31.115.195:37339): tablet dd89830cf3a24f179a73fb2db057ac4e (table test-table [id=911c8aa7053d4e42b5c18e1c1e945418]) successfully deleted
I20260620 08:08:19.257347 18355 catalog_manager.cc:4841] TS 5fe5e05e8b844e2bb285c23d394d32d1 (127.31.115.194:40967): tablet dd89830cf3a24f179a73fb2db057ac4e (table test-table [id=911c8aa7053d4e42b5c18e1c1e945418]) successfully deleted
W20260620 08:08:19.259202 18356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:41637 (error 108)
I20260620 08:08:19.259397 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18562
I20260620 08:08:19.260998 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:19.260826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:41637 (local address 127.0.0.1:40108)
0620 08:08:19.260899 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:19.260954 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:41637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:08:19.261163 18356 catalog_manager.cc:4584] TS fdeea12291b644eba373ec9b825b2faa (127.31.115.193:41637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd89830cf3a24f179a73fb2db057ac4e: Network error: Client connection negotiation failed: client connection to 127.31.115.193:41637: connect: Connection refused (error 111)
I20260620 08:08:19.264644 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18693
I20260620 08:08:19.269871 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18340
2026-06-20T08:08:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (745 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:19 dist-test-slave-c4w8 krb5kdc[18864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:19 dist-test-slave-c4w8 krb5kdc[18864](info): set up 2 sockets
Jun 20 08:08:19 dist-test-slave-c4w8 krb5kdc[18864](info): commencing operation
krb5kdc: starting...
W20260620 08:08:20.379105 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:20 dist-test-slave-c4w8 krb5kdc[18864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:20.531579 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34883
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:20.623790 18877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:20.624076 18877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:20.624123 18877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:20.625432 18877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:20.625525 18877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:20.625551 18877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:20.625573 18877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:20.625640 18877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:20.627353 18877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:34883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:20.627671 18877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:20.627944 18877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:20.633530 18882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:20.633524 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:20.633538 18883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:20.634002 18877 server_base.cc:1034] running on GCE node
I20260620 08:08:20.634334 18877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:20.634618 18877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:20.635852 18877 hybrid_clock.cc:648] HybridClock initialized: now 1781942900635830 us; error 31 us; skew 500 ppm
Jun 20 08:08:20 dist-test-slave-c4w8 krb5kdc[18864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:20.645023 18877 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:20.645584 18877 webserver.cc:460] Webserver started at http://127.31.115.254:37445/ using document root <none> and password file <none>
I20260620 08:08:20.645907 18877 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:20.645957 18877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:20.646111 18877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:20.647393 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "473b37b190374f30861d16cb6298f57b"
format_stamp: "Formatted at 2026-06-20 08:08:20 on dist-test-slave-c4w8"
server_key: "16e7ab5f964b911158cdea109c9dd05b"
server_key_iv: "cdfd5bc71eca68a1b52d72b3ba26ac85"
server_key_version: "encryptionkey@0"
I20260620 08:08:20.647801 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "473b37b190374f30861d16cb6298f57b"
format_stamp: "Formatted at 2026-06-20 08:08:20 on dist-test-slave-c4w8"
server_key: "16e7ab5f964b911158cdea109c9dd05b"
server_key_iv: "cdfd5bc71eca68a1b52d72b3ba26ac85"
server_key_version: "encryptionkey@0"
I20260620 08:08:20.649318 18877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:20.650650 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:20.650997 18877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:20.651124 18877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "473b37b190374f30861d16cb6298f57b"
format_stamp: "Formatted at 2026-06-20 08:08:20 on dist-test-slave-c4w8"
server_key: "16e7ab5f964b911158cdea109c9dd05b"
server_key_iv: "cdfd5bc71eca68a1b52d72b3ba26ac85"
server_key_version: "encryptionkey@0"
I20260620 08:08:20.651234 18877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:20.666793 18877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:20.669762 18877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:20.670172 18877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:20.674813 18877 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:34883
I20260620 08:08:20.674880 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:34883 every 8 connection(s)
I20260620 08:08:20.675292 18877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:20.676412 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:20.677578 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18877
I20260620 08:08:20.677807 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:20.678136 32207 external_mini_cluster.cc:1428] Setting key 3ccd8175bc61bb3b72e7c03ab6b7fa71
I20260620 08:08:20.679510 18945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: Bootstrap starting.
I20260620 08:08:20.680168 18945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:20.680488 18945 log.cc:824] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:20.681248 18945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: No bootstrap required, opened a new log
I20260620 08:08:20.683524 18945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473b37b190374f30861d16cb6298f57b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34883 } }
I20260620 08:08:20.683709 18945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:20.683760 18945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473b37b190374f30861d16cb6298f57b, State: Initialized, Role: FOLLOWER
I20260620 08:08:20.683966 18945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473b37b190374f30861d16cb6298f57b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34883 } }
I20260620 08:08:20.684073 18945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:20.684115 18945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:20.684163 18945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:20.685079 18945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473b37b190374f30861d16cb6298f57b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34883 } }
I20260620 08:08:20.685214 18945 leader_election.cc:302] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473b37b190374f30861d16cb6298f57b; no voters:
I20260620 08:08:20.685413 18945 leader_election.cc:288] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:20.685443 18950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:20.685559 18950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [term 1 LEADER]: Becoming Leader. State: Replica: 473b37b190374f30861d16cb6298f57b, State: Running, Role: LEADER
I20260620 08:08:20.685683 18950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473b37b190374f30861d16cb6298f57b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34883 } }
I20260620 08:08:20.685735 18945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 08:08:20 dist-test-slave-c4w8 krb5kdc[18864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:20.686986 18950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473b37b190374f30861d16cb6298f57b. Latest consensus state: current_term: 1 leader_uuid: "473b37b190374f30861d16cb6298f57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473b37b190374f30861d16cb6298f57b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34883 } } }
I20260620 08:08:20.687084 18950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:20.687526 18951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473b37b190374f30861d16cb6298f57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473b37b190374f30861d16cb6298f57b" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 34883 } } }
I20260620 08:08:20.687889 18951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b [sys.catalog]: This master's current role is: LEADER
W20260620 08:08:20.687711 18964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:20.688076 18964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:20.688154 18965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:20.688354 18965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:20.690979 18965 catalog_manager.cc:1347] Generated new cluster ID: bfad42b3f755426ea24c5eb3886bc609
I20260620 08:08:20.691041 18965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:20.691666 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:20.678973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.31.115.254:34883)
0620 08:08:20.679182 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:20.679190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:20.679216 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:08:20.679654 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:20.679663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:20.679728 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:20.679895 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:20.680647 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:20.681917 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:20.682825 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:20.683273 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:20.683412 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:20.686530 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:20.686536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:20.686536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:20.686554 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:20.689215 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:20.689710 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:20.689714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:20.689714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:20.689835 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:20.690187 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:20.690188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:20.690189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:20.690531 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:20.690697 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:20.691291 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 08:08:20.691439 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260620 08:08:20.704707 18965 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:20.705497 18965 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:20.720171 18965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: Generated new TSK 0
I20260620 08:08:20.720494 18965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.115.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
Entry for principal kudu/127.31.115.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab.
I20260620 08:08:20.759753 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34883
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:20.852334 18971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:20.852618 18971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:20.852661 18971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:20.853971 18971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:20.854066 18971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:20.854117 18971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:08:20.856282 18971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.193.keytab
--principal=kudu/127.31.115.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:20.856634 18971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:20.856900 18971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:20.863606 18977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:20.863621 18979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:20.863736 18971 server_base.cc:1034] running on GCE node
W20260620 08:08:20.864596 18976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:20.865039 18971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:20.865347 18971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:20.866647 18971 hybrid_clock.cc:648] HybridClock initialized: now 1781942900866633 us; error 39 us; skew 500 ppm
Jun 20 08:08:20 dist-test-slave-c4w8 krb5kdc[18864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:20.869884 18971 init.cc:377] Logged in from keytab as kudu/127.31.115.193@KRBTEST.COM (short username kudu)
I20260620 08:08:20.870661 18971 webserver.cc:460] Webserver started at http://127.31.115.193:46153/ using document root <none> and password file <none>
I20260620 08:08:20.871176 18971 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:20.871419 18971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:20.871718 18971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:20.872910 18971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "8be8b3b3f5004270b79487e112aff70e"
format_stamp: "Formatted at 2026-06-20 08:08:20 on dist-test-slave-c4w8"
server_key: "b8c3e0b5e8fc195de78c1d45204aeb51"
server_key_iv: "d7732afb552389422db81bf17d99e165"
server_key_version: "encryptionkey@0"
I20260620 08:08:20.873306 18971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "8be8b3b3f5004270b79487e112aff70e"
format_stamp: "Formatted at 2026-06-20 08:08:20 on dist-test-slave-c4w8"
server_key: "b8c3e0b5e8fc195de78c1d45204aeb51"
server_key_iv: "d7732afb552389422db81bf17d99e165"
server_key_version: "encryptionkey@0"
I20260620 08:08:20.875108 18971 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 08:08:20.876294 18986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:20.876657 18971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:20.876824 18971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "8be8b3b3f5004270b79487e112aff70e"
format_stamp: "Formatted at 2026-06-20 08:08:20 on dist-test-slave-c4w8"
server_key: "b8c3e0b5e8fc195de78c1d45204aeb51"
server_key_iv: "d7732afb552389422db81bf17d99e165"
server_key_version: "encryptionkey@0"
I20260620 08:08:20.876945 18971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:20.906896 18971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:20.910143 18971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:20.910778 18971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:20.911365 18971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:20.911772 18971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:20.911849 18971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:20.911890 18971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:20.911921 18971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:20.922672 18971 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:36521
I20260620 08:08:20.923094 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:36521 every 8 connection(s)
I20260620 08:08:20.923245 18971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:08:20.927122 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 18971
I20260620 08:08:20.927220 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:08:20.927539 32207 external_mini_cluster.cc:1428] Setting key 92e9ca9fc2d63377cda6376f0a60c17b
Jun 20 08:08:20 dist-test-slave-c4w8 krb5kdc[18864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.193@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:20.937659 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:20.924524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:52805 (local address 127.31.115.254:34883)
0620 08:08:20.924650 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:20.924652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:20.924941 (+ 289us) server_negotiation.cc:407] Connection header received
0620 08:08:20.925510 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:20.925512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:20.925527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:20.925581 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:20.926883 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:20.927466 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:20.929491 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:20.929763 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:20.929850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:20.933897 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:20.933899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:20.933900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:20.933910 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:20.935675 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:20.936217 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:20.936219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:20.936220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:20.936287 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:20.936589 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:20.936590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:20.936590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:20.936733 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:20.936824 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:20.937401 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 08:08:20.937509 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 08:08:20.937880 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:20.924431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34883 (local address 127.31.115.193:52805)
0620 08:08:20.924687 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:20.924809 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:20.925385 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:20.925687 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:20.925691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:20.926475 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:20.926701 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:20.926706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:20.927573 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:20.927574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:20.929223 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:20.929227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:20.929866 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:20.929868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:20.930014 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:20.930886 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:20.930899 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:20.933750 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:20.935821 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:20.935835 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:20.935838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:20.936120 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:20.936370 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:20.936371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:20.936371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:20.936504 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:20.936809 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:20.936818 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:20.936984 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 08:08:20.937372 (+ 388us) client_negotiation.cc:240] Negotiation successful
0620 08:08:20.937579 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
I20260620 08:08:20.938625 19100 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34883
I20260620 08:08:20.938750 19100 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:20.938978 19100 heartbeater.cc:511] Master 127.31.115.254:34883 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.115.194@KRBTEST.COM; defaulting to no policy
I20260620 08:08:20.939725 18909 ts_manager.cc:194] Registered new tserver with Master: 8be8b3b3f5004270b79487e112aff70e (127.31.115.193:36521)
I20260620 08:08:20.940698 18909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.193@KRBTEST.COM'} at 127.31.115.193:52805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
Entry for principal kudu/127.31.115.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab.
I20260620 08:08:20.973106 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34883
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:21.087487 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:21.087735 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:21.087984 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:21.089673 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:21.089730 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:21.089771 19105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:08:21.092190 19105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.194.keytab
--principal=kudu/127.31.115.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:21.092550 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:21.092856 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:21.100014 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:21.100003 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:21.100003 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:21.101243 19105 server_base.cc:1034] running on GCE node
I20260620 08:08:21.101477 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:21.102005 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:21.103240 19105 hybrid_clock.cc:648] HybridClock initialized: now 1781942901103102 us; error 182 us; skew 500 ppm
Jun 20 08:08:21 dist-test-slave-c4w8 krb5kdc[18864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:21.106804 19105 init.cc:377] Logged in from keytab as kudu/127.31.115.194@KRBTEST.COM (short username kudu)
I20260620 08:08:21.107252 19105 webserver.cc:460] Webserver started at http://127.31.115.194:40177/ using document root <none> and password file <none>
I20260620 08:08:21.107463 19105 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:21.107501 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:21.107649 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:21.108860 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "d5ffe408707d4aaab20074528e4f51b7"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "1b7aa804e5319983393bde2f0b355134"
server_key_iv: "0d255f8e07c6517c1fd74cdd83aee2dd"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.109185 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "d5ffe408707d4aaab20074528e4f51b7"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "1b7aa804e5319983393bde2f0b355134"
server_key_iv: "0d255f8e07c6517c1fd74cdd83aee2dd"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.110911 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 08:08:21.112222 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.112646 19105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:21.112766 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "d5ffe408707d4aaab20074528e4f51b7"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "1b7aa804e5319983393bde2f0b355134"
server_key_iv: "0d255f8e07c6517c1fd74cdd83aee2dd"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.112836 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:21.142146 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:21.144567 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:21.144697 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:21.144924 19105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:21.145203 19105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:21.145233 19105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.145257 19105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:21.145269 19105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.151968 19105 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:41699
I20260620 08:08:21.152061 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:41699 every 8 connection(s)
I20260620 08:08:21.152555 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:08:21.161028 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19105
I20260620 08:08:21.161131 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
Jun 20 08:08:21 dist-test-slave-c4w8 krb5kdc[18864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.194@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:21.161433 32207 external_mini_cluster.cc:1428] Setting key 3150822ecf1bb3a91311f405211f7b1e
I20260620 08:08:21.167199 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.153838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:40919 (local address 127.31.115.254:34883)
0620 08:08:21.153964 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.153972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.154023 (+ 51us) server_negotiation.cc:407] Connection header received
0620 08:08:21.154646 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.154648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.154663 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.154737 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:21.156279 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.156933 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.158473 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.158672 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.158742 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.162159 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:21.162161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:21.162162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:21.162171 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:21.164427 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:21.165421 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:21.165422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:21.165422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:21.165540 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:21.166120 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:21.166121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:21.166121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:21.166259 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:21.166317 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.167036 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.167126 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 08:08:21.167469 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.153699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34883 (local address 127.31.115.194:40919)
0620 08:08:21.153870 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.153920 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.154412 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.154876 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.154880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.155628 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:21.156045 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.156051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.157077 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.157078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.158351 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.158354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.158872 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.158873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.159064 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.159960 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:21.159985 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:21.161933 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:21.164650 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:21.164663 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:21.164667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:21.165087 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:21.165667 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:21.165669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:21.165669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:21.165984 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:21.166339 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:21.166347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:21.166709 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.166955 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.167174 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260620 08:08:21.168099 19234 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34883
I20260620 08:08:21.168269 19234 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:21.168591 19234 heartbeater.cc:511] Master 127.31.115.254:34883 requested a full tablet report, sending...
I20260620 08:08:21.169340 18909 ts_manager.cc:194] Registered new tserver with Master: d5ffe408707d4aaab20074528e4f51b7 (127.31.115.194:41699)
I20260620 08:08:21.169903 18909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.194@KRBTEST.COM'} at 127.31.115.194:40919
WARNING: no policy specified for kudu/127.31.115.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
Entry for principal kudu/127.31.115.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab.
I20260620 08:08:21.210608 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34883
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:21.322131 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:21.322373 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:21.322402 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:21.323832 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:21.323891 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:21.323922 19239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:08:21.325984 19239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.195.keytab
--principal=kudu/127.31.115.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:34883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:21.326274 19239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:21.326509 19239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:21.333842 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:21.333774 19239 server_base.cc:1034] running on GCE node
W20260620 08:08:21.333813 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:21.333813 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:21.334590 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:21.334952 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:21.336138 19239 hybrid_clock.cc:648] HybridClock initialized: now 1781942901336078 us; error 81 us; skew 500 ppm
Jun 20 08:08:21 dist-test-slave-c4w8 krb5kdc[18864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:21.340066 19239 init.cc:377] Logged in from keytab as kudu/127.31.115.195@KRBTEST.COM (short username kudu)
I20260620 08:08:21.340695 19239 webserver.cc:460] Webserver started at http://127.31.115.195:46445/ using document root <none> and password file <none>
I20260620 08:08:21.341099 19239 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:21.341189 19239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:21.341482 19239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:21.342689 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "0171ada13b324bb080ef794afbd7f8db"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "9bc21542f5bccddb818bbcb306c45ea3"
server_key_iv: "7021cc3877ae1f995196cf4fa64dc30f"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.343111 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "0171ada13b324bb080ef794afbd7f8db"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "9bc21542f5bccddb818bbcb306c45ea3"
server_key_iv: "7021cc3877ae1f995196cf4fa64dc30f"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.344866 19239 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:08:21.346028 19254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.346470 19239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:21.346812 19239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "0171ada13b324bb080ef794afbd7f8db"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "9bc21542f5bccddb818bbcb306c45ea3"
server_key_iv: "7021cc3877ae1f995196cf4fa64dc30f"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.346928 19239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:21.369068 19239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:21.372092 19239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:21.372291 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:21.372576 19239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:21.372946 19239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:21.373028 19239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.373075 19239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:21.373102 19239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.381772 19239 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:39173
I20260620 08:08:21.381861 19367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:39173 every 8 connection(s)
I20260620 08:08:21.382591 19239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:08:21.387518 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19239
I20260620 08:08:21.387626 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:08:21.387850 32207 external_mini_cluster.cc:1428] Setting key b1e83f68df96e7f1aba196992cee7489
Jun 20 08:08:21 dist-test-slave-c4w8 krb5kdc[18864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.195@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:21.397369 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.383998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:38667 (local address 127.31.115.254:34883)
0620 08:08:21.384079 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.384082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.384377 (+ 295us) server_negotiation.cc:407] Connection header received
0620 08:08:21.384891 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.384893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.384905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.384982 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:21.386605 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.387368 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.389190 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.389400 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.389478 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.393377 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:21.393378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:21.393379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:21.393387 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:21.395037 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:21.395726 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:21.395727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:21.395728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:21.395796 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:21.396186 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:21.396187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:21.396188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:21.396323 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:21.396397 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.397097 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.397235 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:08:21.397656 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.383990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:34883 (local address 127.31.115.195:38667)
0620 08:08:21.384229 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.384268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.384751 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.385094 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.385098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.386207 (+ 1109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:21.386436 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.386443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.387521 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.387522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.389047 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.389056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.389845 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.389847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.390245 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.390915 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:21.390925 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:21.393189 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:21.395213 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:21.395224 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:21.395226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:21.395606 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:21.395922 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:21.395923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:21.395923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:21.396070 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:21.396485 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:21.396492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:21.396819 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.396972 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.397200 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":172,"threads_started":1}
I20260620 08:08:21.398540 19368 heartbeater.cc:347] Connected to a master server at 127.31.115.254:34883
I20260620 08:08:21.398669 19368 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:21.398897 19368 heartbeater.cc:511] Master 127.31.115.254:34883 requested a full tablet report, sending...
I20260620 08:08:21.399818 18909 ts_manager.cc:194] Registered new tserver with Master: 0171ada13b324bb080ef794afbd7f8db (127.31.115.195:39173)
I20260620 08:08:21.400635 18909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.115.195@KRBTEST.COM'} at 127.31.115.195:38667
I20260620 08:08:21.400750 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:08:21.411720 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.402850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.31.115.254:34883)
0620 08:08:21.402980 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.402982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.403068 (+ 86us) server_negotiation.cc:407] Connection header received
0620 08:08:21.403153 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.403154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.403165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.403226 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:21.403850 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.404456 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.405266 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.405464 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.405556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.407748 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:21.407751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:21.407751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:21.407762 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:21.410166 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:21.410652 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:21.410653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:21.410654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:21.410733 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:21.410983 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:21.410984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:21.410984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:21.411184 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:21.411323 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.411490 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.411587 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 08:08:21.414732 18909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:08:21.417451 18909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:08:21.425431 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.420943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:39173 (local address 127.0.0.1:50602)
0620 08:08:21.421348 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.421358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.421398 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.421715 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.421719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.421722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.421806 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.421809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.423170 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.423171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.424315 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.424317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.425220 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.425221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.425280 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.425283 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.425328 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.425361 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":5,"thread_start_us":118,"threads_started":1}
I20260620 08:08:21.425431 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.420946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:36521 (local address 127.0.0.1:53528)
0620 08:08:21.421485 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.421493 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.421536 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.421870 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.421871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.421873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.421927 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.421929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.422895 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.422896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.424362 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.424364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.425088 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.425089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.425173 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.425187 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.425257 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.425318 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":7,"spinlock_wait_cycles":768,"thread_start_us":109,"threads_started":1}
I20260620 08:08:21.425645 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.421218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.31.115.193:36521)
0620 08:08:21.421355 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.421360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.421601 (+ 241us) server_negotiation.cc:407] Connection header received
0620 08:08:21.421667 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.421669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.421686 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.421787 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.422011 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.422775 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.424450 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.424983 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.425078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.425142 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.425474 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.425527 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260620 08:08:21.425484 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.421227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.31.115.195:39173)
0620 08:08:21.421424 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.421429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.421440 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:21.421484 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.421487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.421507 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.421582 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.421897 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.423057 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.424407 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.425144 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.425223 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.425274 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.425353 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.425390 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260620 08:08:21.426515 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.420927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41699 (local address 127.0.0.1:53088)
0620 08:08:21.421113 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.421144 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.421235 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.422031 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.422032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.422035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.422070 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.422071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.423412 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.423412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.424345 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.424347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.426187 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.426188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.426329 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.426333 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.426390 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.426434 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260620 08:08:21.426415 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 (DEFAULT_TABLE table=test-table [id=cb6be15e7bd34b26b613240e8e28964e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:21.426564 19302 tablet_service.cc:1467] Processing CreateTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 (DEFAULT_TABLE table=test-table [id=cb6be15e7bd34b26b613240e8e28964e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:21.426832 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.421426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.31.115.194:41699)
0620 08:08:21.421596 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.421601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.421631 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:08:21.421742 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.421747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.421766 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.422132 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.422210 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.423302 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.424461 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.425145 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.425223 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.425274 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.426694 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.426751 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260620 08:08:21.426851 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0404e05c4aa74e0fbb1109b622fd2588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:21.426913 19302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0404e05c4aa74e0fbb1109b622fd2588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:21.427814 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 (DEFAULT_TABLE table=test-table [id=cb6be15e7bd34b26b613240e8e28964e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:21.428182 19168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0404e05c4aa74e0fbb1109b622fd2588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:21.432427 19387 tablet_bootstrap.cc:492] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Bootstrap starting.
I20260620 08:08:21.432792 19385 tablet_bootstrap.cc:492] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Bootstrap starting.
I20260620 08:08:21.432632 19386 tablet_bootstrap.cc:492] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: Bootstrap starting.
I20260620 08:08:21.433203 19387 tablet_bootstrap.cc:654] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:21.433485 19385 tablet_bootstrap.cc:654] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:21.433840 19387 log.cc:824] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:21.434111 19386 tablet_bootstrap.cc:654] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:21.434144 19385 log.cc:824] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:21.434666 19387 tablet_bootstrap.cc:492] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: No bootstrap required, opened a new log
I20260620 08:08:21.434751 19387 ts_tablet_manager.cc:1397] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:08:21.434906 19385 tablet_bootstrap.cc:492] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: No bootstrap required, opened a new log
I20260620 08:08:21.434973 19385 ts_tablet_manager.cc:1397] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:08:21.435089 19386 log.cc:824] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:21.436480 19387 raft_consensus.cc:348] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.436668 19387 raft_consensus.cc:374] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:21.436713 19387 raft_consensus.cc:729] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ffe408707d4aaab20074528e4f51b7, State: Initialized, Role: FOLLOWER
I20260620 08:08:21.436830 19387 consensus_queue.cc:260] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.437175 19387 ts_tablet_manager.cc:1428] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:08:21.437224 19234 heartbeater.cc:503] Master 127.31.115.254:34883 was elected leader, sending a full tablet report...
I20260620 08:08:21.437100 19385 raft_consensus.cc:348] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.437271 19385 raft_consensus.cc:374] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:21.437304 19385 raft_consensus.cc:729] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0171ada13b324bb080ef794afbd7f8db, State: Initialized, Role: FOLLOWER
I20260620 08:08:21.437413 19385 consensus_queue.cc:260] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.437728 19385 ts_tablet_manager.cc:1428] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 08:08:21.437830 19368 heartbeater.cc:503] Master 127.31.115.254:34883 was elected leader, sending a full tablet report...
I20260620 08:08:21.437933 19386 tablet_bootstrap.cc:492] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: No bootstrap required, opened a new log
I20260620 08:08:21.438000 19386 ts_tablet_manager.cc:1397] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:08:21.439841 19386 raft_consensus.cc:348] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.440007 19386 raft_consensus.cc:374] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:21.440059 19386 raft_consensus.cc:729] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be8b3b3f5004270b79487e112aff70e, State: Initialized, Role: FOLLOWER
I20260620 08:08:21.440184 19386 consensus_queue.cc:260] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.440455 19386 ts_tablet_manager.cc:1428] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:08:21.440464 19100 heartbeater.cc:503] Master 127.31.115.254:34883 was elected leader, sending a full tablet report...
I20260620 08:08:21.616266 19391 raft_consensus.cc:482] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:21.616449 19391 raft_consensus.cc:504] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.617198 19391 leader_election.cc:288] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8be8b3b3f5004270b79487e112aff70e (127.31.115.193:36521), 0171ada13b324bb080ef794afbd7f8db (127.31.115.195:39173)
I20260620 08:08:21.619346 19393 raft_consensus.cc:482] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:21.619483 19393 raft_consensus.cc:504] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.620218 19393 leader_election.cc:288] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5ffe408707d4aaab20074528e4f51b7 (127.31.115.194:41699), 0171ada13b324bb080ef794afbd7f8db (127.31.115.195:39173)
I20260620 08:08:21.621083 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.617573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:58855 (local address 127.31.115.195:39173)
0620 08:08:21.617914 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.617916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.617925 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:08:21.617960 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.617961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.617974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.618075 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.618273 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.618777 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.620076 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.620670 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.620723 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.620734 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.620971 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.621011 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260620 08:08:21.621106 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.617316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:36521 (local address 127.31.115.194:50233)
0620 08:08:21.617418 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.617432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.617501 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.617798 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.617799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.617801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.617891 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.617893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.618897 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.618898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.620038 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.620042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.620810 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.620811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.620906 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.620910 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.620959 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.621011 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":56}
I20260620 08:08:21.621315 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.617436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:50233 (local address 127.31.115.193:36521)
0620 08:08:21.617535 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.617538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.617550 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:08:21.617585 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.617586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.617597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.617679 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.618156 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.618759 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.620139 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.620716 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.620812 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.620821 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.621194 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.621255 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 08:08:21.621320 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.617389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:39173 (local address 127.31.115.194:58855)
0620 08:08:21.617574 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.617584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.617879 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.618107 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.618108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.618110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.618192 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.618193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.618876 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.618878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.619974 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.619978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.620808 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.620809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.620888 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.620892 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.621176 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.621241 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260620 08:08:21.621733 19320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0404e05c4aa74e0fbb1109b622fd2588" candidate_uuid: "d5ffe408707d4aaab20074528e4f51b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0171ada13b324bb080ef794afbd7f8db" is_pre_election: true
I20260620 08:08:21.621757 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0404e05c4aa74e0fbb1109b622fd2588" candidate_uuid: "d5ffe408707d4aaab20074528e4f51b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be8b3b3f5004270b79487e112aff70e" is_pre_election: true
I20260620 08:08:21.621866 19054 raft_consensus.cc:2454] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ffe408707d4aaab20074528e4f51b7 in term 0.
I20260620 08:08:21.621903 19320 raft_consensus.cc:2454] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ffe408707d4aaab20074528e4f51b7 in term 0.
I20260620 08:08:21.622076 19124 leader_election.cc:302] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be8b3b3f5004270b79487e112aff70e, d5ffe408707d4aaab20074528e4f51b7; no voters:
I20260620 08:08:21.622215 19391 raft_consensus.cc:2781] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:08:21.622270 19391 raft_consensus.cc:482] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:08:21.622298 19391 raft_consensus.cc:3037] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:21.623344 19391 raft_consensus.cc:504] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.623631 19054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0404e05c4aa74e0fbb1109b622fd2588" candidate_uuid: "d5ffe408707d4aaab20074528e4f51b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be8b3b3f5004270b79487e112aff70e"
I20260620 08:08:21.623718 19054 raft_consensus.cc:3037] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:21.624070 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.620243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:46575 (local address 127.31.115.194:41699)
0620 08:08:21.620283 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.620284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.620936 (+ 652us) server_negotiation.cc:407] Connection header received
0620 08:08:21.621030 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.621030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.621039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.621066 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.621269 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.621982 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.623151 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.623632 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.623734 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.623743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.623972 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.624007 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:08:21.624100 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.620037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:41699 (local address 127.31.115.193:46575)
0620 08:08:21.620884 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.620893 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.620962 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.621126 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.621127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.621129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.621194 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.621195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.622053 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.622054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.623000 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.623003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.623786 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.623787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.623872 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.623876 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.623950 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.623995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":112,"threads_started":1}
I20260620 08:08:21.624547 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.620023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:39173 (local address 127.31.115.193:40473)
0620 08:08:21.620354 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.620365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:21.620415 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:21.620648 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:21.620650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:21.620653 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:21.620728 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.620730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.622453 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.622454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.623576 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:21.623578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.624353 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.624355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:21.624413 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.624417 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:21.624466 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:08:21.624498 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":100,"threads_started":1}
I20260620 08:08:21.624709 19054 raft_consensus.cc:2454] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ffe408707d4aaab20074528e4f51b7 in term 1.
I20260620 08:08:21.624756 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.620337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:40473 (local address 127.31.115.195:39173)
0620 08:08:21.620509 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.620511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.620517 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:08:21.620538 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.620539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.620548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.620613 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:21.620814 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.622246 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.623665 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.624220 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.624349 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.624358 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.624501 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.624548 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260620 08:08:21.624881 19124 leader_election.cc:302] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be8b3b3f5004270b79487e112aff70e, d5ffe408707d4aaab20074528e4f51b7; no voters:
I20260620 08:08:21.624961 19391 leader_election.cc:288] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8be8b3b3f5004270b79487e112aff70e (127.31.115.193:36521), 0171ada13b324bb080ef794afbd7f8db (127.31.115.195:39173)
I20260620 08:08:21.625010 19391 raft_consensus.cc:2781] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:21.624984 19320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0404e05c4aa74e0fbb1109b622fd2588" candidate_uuid: "8be8b3b3f5004270b79487e112aff70e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0171ada13b324bb080ef794afbd7f8db" is_pre_election: true
I20260620 08:08:21.625049 19391 raft_consensus.cc:686] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 1 LEADER]: Becoming Leader. State: Replica: d5ffe408707d4aaab20074528e4f51b7, State: Running, Role: LEADER
I20260620 08:08:21.625074 19320 raft_consensus.cc:2454] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be8b3b3f5004270b79487e112aff70e in term 0.
I20260620 08:08:21.625125 19322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0404e05c4aa74e0fbb1109b622fd2588" candidate_uuid: "d5ffe408707d4aaab20074528e4f51b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0171ada13b324bb080ef794afbd7f8db"
I20260620 08:08:21.625134 19391 consensus_queue.cc:237] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } }
I20260620 08:08:21.625198 19322 raft_consensus.cc:3037] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:21.625228 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0404e05c4aa74e0fbb1109b622fd2588" candidate_uuid: "8be8b3b3f5004270b79487e112aff70e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ffe408707d4aaab20074528e4f51b7" is_pre_election: true
I20260620 08:08:21.625272 18988 leader_election.cc:302] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0171ada13b324bb080ef794afbd7f8db, 8be8b3b3f5004270b79487e112aff70e; no voters:
I20260620 08:08:21.625725 19393 raft_consensus.cc:2741] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 08:08:21.626091 18909 catalog_manager.cc:5495] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5ffe408707d4aaab20074528e4f51b7 (127.31.115.194). New cstate: current_term: 1 leader_uuid: "d5ffe408707d4aaab20074528e4f51b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ffe408707d4aaab20074528e4f51b7" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 41699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 } health_report { overall_health: UNKNOWN } } }
I20260620 08:08:21.626187 19322 raft_consensus.cc:2454] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ffe408707d4aaab20074528e4f51b7 in term 1.
W20260620 08:08:21.633522 19369 tablet.cc:2369] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:08:21.653503 19235 tablet.cc:2369] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:08:21.656919 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.653723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.31.115.194:41699)
0620 08:08:21.653776 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.653778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.653922 (+ 144us) server_negotiation.cc:407] Connection header received
0620 08:08:21.653959 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.653960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.653976 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.654034 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:21.654342 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.655054 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.655987 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.656187 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.656254 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.656476 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:21.656584 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:21.656656 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.656790 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.656814 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:21.659746 19054 raft_consensus.cc:1261] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 1 FOLLOWER]: Refusing update from remote peer d5ffe408707d4aaab20074528e4f51b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:21.659786 19322 raft_consensus.cc:1261] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 1 FOLLOWER]: Refusing update from remote peer d5ffe408707d4aaab20074528e4f51b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:21.660127 19398 consensus_queue.cc:1035] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8be8b3b3f5004270b79487e112aff70e" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:21.660269 19391 consensus_queue.cc:1035] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0171ada13b324bb080ef794afbd7f8db" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 39173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:21.663417 19401 mvcc.cc:205] Tried to move back new op lower bound from 7298838125193469952 to 7298838125057458176. Current Snapshot: MvccSnapshot[applied={T|T < 7298838125193469952}]
I20260620 08:08:21.664239 19402 mvcc.cc:205] Tried to move back new op lower bound from 7298838125193469952 to 7298838125057458176. Current Snapshot: MvccSnapshot[applied={T|T < 7298838125193469952}]
I20260620 08:08:21.666064 18909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:21.666193 18909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:21.667434 18909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 473b37b190374f30861d16cb6298f57b: Sending DeleteTablet for 3 replicas of tablet 0404e05c4aa74e0fbb1109b622fd2588
I20260620 08:08:21.667757 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:21 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53528
I20260620 08:08:21.667807 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:21 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53088
I20260620 08:08:21.668076 19412 tablet_replica.cc:330] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: stopping tablet replica
I20260620 08:08:21.668167 19411 tablet_replica.cc:330] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e: stopping tablet replica
I20260620 08:08:21.668279 19411 raft_consensus.cc:2229] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:21.668277 19412 raft_consensus.cc:2229] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:08:21.668412 19411 raft_consensus.cc:2258] T 0404e05c4aa74e0fbb1109b622fd2588 P 8be8b3b3f5004270b79487e112aff70e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:21.668404 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18971
I20260620 08:08:21.668509 19412 raft_consensus.cc:2258] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:21.669412 19412 ts_tablet_manager.cc:1905] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:21.671375 19412 ts_tablet_manager.cc:1918] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:21.671499 19412 log.cc:1196] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/0404e05c4aa74e0fbb1109b622fd2588
I20260620 08:08:21.671813 19412 ts_tablet_manager.cc:1939] T 0404e05c4aa74e0fbb1109b622fd2588 P d5ffe408707d4aaab20074528e4f51b7: Deleting consensus metadata
I20260620 08:08:21.674285 18893 catalog_manager.cc:4841] TS d5ffe408707d4aaab20074528e4f51b7 (127.31.115.194:41699): tablet 0404e05c4aa74e0fbb1109b622fd2588 (table test-table [id=cb6be15e7bd34b26b613240e8e28964e]) successfully deleted
I20260620 08:08:21.674990 19300 tablet_service.cc:1514] Processing DeleteTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:21 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50602
I20260620 08:08:21.674978 19302 tablet_service.cc:1514] Processing DeleteTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:21 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:50602
I20260620 08:08:21.675294 19413 tablet_replica.cc:330] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: stopping tablet replica
I20260620 08:08:21.675438 19413 raft_consensus.cc:2229] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:21.675595 19413 raft_consensus.cc:2258] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:21.678064 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 0404e05c4aa74e0fbb1109b622fd2588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:21 UTC) from {username='kudu', principal='kudu/127.31.115.254@KRBTEST.COM'} at 127.0.0.1:53088
W20260620 08:08:21.678476 18893 catalog_manager.cc:4808] TS d5ffe408707d4aaab20074528e4f51b7 (127.31.115.194:41699): delete failed for tablet 0404e05c4aa74e0fbb1109b622fd2588 because the tablet was not found. No further retry: Not found: Tablet not found: 0404e05c4aa74e0fbb1109b622fd2588
I20260620 08:08:21.678529 19413 ts_tablet_manager.cc:1905] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:08:21.680330 18896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:36521 (error 108)
I20260620 08:08:21.680615 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19105
I20260620 08:08:21.681671 19413 ts_tablet_manager.cc:1918] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:21.681686 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.681416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:36521 (local address 127.0.0.1:53534)
0620 08:08:21.681579 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:21.681643 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:36521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 08:08:21.681756 19413 log.cc:1196] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/0404e05c4aa74e0fbb1109b622fd2588
W20260620 08:08:21.682181 18896 catalog_manager.cc:4584] TS 8be8b3b3f5004270b79487e112aff70e (127.31.115.193:36521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0404e05c4aa74e0fbb1109b622fd2588: Network error: Client connection negotiation failed: client connection to 127.31.115.193:36521: connect: Connection refused (error 111)
I20260620 08:08:21.682253 19413 ts_tablet_manager.cc:1939] T 0404e05c4aa74e0fbb1109b622fd2588 P 0171ada13b324bb080ef794afbd7f8db: Deleting consensus metadata
I20260620 08:08:21.682797 18894 catalog_manager.cc:4841] TS 0171ada13b324bb080ef794afbd7f8db (127.31.115.195:39173): tablet 0404e05c4aa74e0fbb1109b622fd2588 (table test-table [id=cb6be15e7bd34b26b613240e8e28964e]) successfully deleted
W20260620 08:08:21.682909 18894 catalog_manager.cc:4808] TS 0171ada13b324bb080ef794afbd7f8db (127.31.115.195:39173): delete failed for tablet 0404e05c4aa74e0fbb1109b622fd2588 because the tablet was not found. No further retry: Not found: Tablet not found: 0404e05c4aa74e0fbb1109b622fd2588
I20260620 08:08:21.686908 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19239
I20260620 08:08:21.692704 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 18877
2026-06-20T08:08:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2423 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T08:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:21Z Disabled control of system clock
I20260620 08:08:21.723035 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:45319
--webserver_interface=127.31.115.254
--webserver_port=0
--builtin_ntp_servers=127.31.115.212:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:45319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:21.814244 19418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:21.814499 19418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:21.814523 19418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:21.815687 19418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:21.815729 19418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:21.815742 19418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:21.815755 19418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:21.815768 19418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:21.817127 19418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:45319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:45319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:21.817356 19418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:21.817711 19418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:21.824159 19424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:21.824208 19426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:21.824419 19423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:21.824718 19418 server_base.cc:1034] running on GCE node
I20260620 08:08:21.825007 19418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:21.825279 19418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:21.826475 19418 hybrid_clock.cc:648] HybridClock initialized: now 1781942901826420 us; error 103 us; skew 500 ppm
I20260620 08:08:21.827831 19418 webserver.cc:460] Webserver started at http://127.31.115.254:41739/ using document root <none> and password file <none>
I20260620 08:08:21.828052 19418 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:21.828095 19418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:21.828188 19418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:21.829380 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "6d41be4a28884ae0aa5a94be327ae825"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "747005b94d444fc5be76519fe01b4e47"
server_key_iv: "ba3649c4f5d291dc627454eb505f45b4"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.829811 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "6d41be4a28884ae0aa5a94be327ae825"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "747005b94d444fc5be76519fe01b4e47"
server_key_iv: "ba3649c4f5d291dc627454eb505f45b4"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.831355 19418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:21.832605 19432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:21.832962 19418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:21.833071 19418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "6d41be4a28884ae0aa5a94be327ae825"
format_stamp: "Formatted at 2026-06-20 08:08:21 on dist-test-slave-c4w8"
server_key: "747005b94d444fc5be76519fe01b4e47"
server_key_iv: "ba3649c4f5d291dc627454eb505f45b4"
server_key_version: "encryptionkey@0"
I20260620 08:08:21.833191 19418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:21.850500 19418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:21.850903 19418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:21.851073 19418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:21.856212 19418 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:45319
I20260620 08:08:21.856405 19484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:45319 every 8 connection(s)
I20260620 08:08:21.856703 19418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:21.858090 19485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:21.859649 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19418
I20260620 08:08:21.859748 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:21.859973 32207 external_mini_cluster.cc:1428] Setting key 5e5a2f93676e65ef945c7bb5ca31646d
I20260620 08:08:21.861086 19485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825: Bootstrap starting.
I20260620 08:08:21.861907 19485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:21.862286 19485 log.cc:824] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:21.863255 19485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825: No bootstrap required, opened a new log
I20260620 08:08:21.866103 19485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41be4a28884ae0aa5a94be327ae825" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 45319 } }
I20260620 08:08:21.866339 19485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:21.866386 19485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d41be4a28884ae0aa5a94be327ae825, State: Initialized, Role: FOLLOWER
I20260620 08:08:21.866601 19485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41be4a28884ae0aa5a94be327ae825" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 45319 } }
I20260620 08:08:21.866706 19485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:21.866734 19485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:21.866818 19485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:21.867344 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:21.860681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.31.115.254:45319)
0620 08:08:21.860941 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:21.860952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:21.860996 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:08:21.861641 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:21.861651 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:21.861739 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:21.861902 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:21.862454 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.863616 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.864789 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:21.865263 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:21.865369 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:21.866489 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:21.866495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:21.866495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:21.866732 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:21.866810 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:21.866829 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:21.866956 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:21.867001 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 08:08:21.867113 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260620 08:08:21.867605 19485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41be4a28884ae0aa5a94be327ae825" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 45319 } }
I20260620 08:08:21.867722 19485 leader_election.cc:302] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d41be4a28884ae0aa5a94be327ae825; no voters:
I20260620 08:08:21.867928 19485 leader_election.cc:288] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:21.867980 19490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:21.868229 19485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:21.868242 19490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [term 1 LEADER]: Becoming Leader. State: Replica: 6d41be4a28884ae0aa5a94be327ae825, State: Running, Role: LEADER
I20260620 08:08:21.868372 19490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41be4a28884ae0aa5a94be327ae825" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 45319 } }
I20260620 08:08:21.868873 19491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d41be4a28884ae0aa5a94be327ae825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41be4a28884ae0aa5a94be327ae825" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 45319 } } }
I20260620 08:08:21.868896 19492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d41be4a28884ae0aa5a94be327ae825. Latest consensus state: current_term: 1 leader_uuid: "6d41be4a28884ae0aa5a94be327ae825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41be4a28884ae0aa5a94be327ae825" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 45319 } } }
I20260620 08:08:21.869104 19492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:21.869060 19491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:21.869769 19500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:21.870286 19500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:21.871995 19500 catalog_manager.cc:1347] Generated new cluster ID: 8573aa87a17d43afbf0746ec3c75cdcd
I20260620 08:08:21.872049 19500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:21.889990 19500 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:21.890641 19500 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:21.893337 19500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825: Generated new TSK 0
I20260620 08:08:21.893599 19500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:21.903899 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.193:0
--local_ip_for_outbound_sockets=127.31.115.193
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:45319
--builtin_ntp_servers=127.31.115.212:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:22.002455 19509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:22.002995 19509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:22.003070 19509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:22.005014 19509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:22.005087 19509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:22.005121 19509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.193
I20260620 08:08:22.007604 19509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.115.193
--webserver_port=0
--tserver_master_addrs=127.31.115.254:45319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.193
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:22.007931 19509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:22.008315 19509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:22.013970 19514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:22.014109 19509 server_base.cc:1034] running on GCE node
W20260620 08:08:22.013970 19515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:22.014272 19517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:22.014711 19509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:22.015049 19509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:22.016360 19509 hybrid_clock.cc:648] HybridClock initialized: now 1781942902016345 us; error 32 us; skew 500 ppm
I20260620 08:08:22.017944 19509 webserver.cc:460] Webserver started at http://127.31.115.193:40325/ using document root <none> and password file <none>
I20260620 08:08:22.018252 19509 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:22.018306 19509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:22.018641 19509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:22.019920 19509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data/instance:
uuid: "ba1e1057b0e54f078b62ac87486c6262"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "d170c8cdc0e84cc168a6602110b17d7a"
server_key_iv: "2a36485d91bd75a8e6f556298c73f72f"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.020318 19509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance:
uuid: "ba1e1057b0e54f078b62ac87486c6262"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "d170c8cdc0e84cc168a6602110b17d7a"
server_key_iv: "2a36485d91bd75a8e6f556298c73f72f"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.022043 19509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 08:08:22.023065 19523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.023485 19509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:22.023617 19509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal
uuid: "ba1e1057b0e54f078b62ac87486c6262"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "d170c8cdc0e84cc168a6602110b17d7a"
server_key_iv: "2a36485d91bd75a8e6f556298c73f72f"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.023720 19509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:22.038444 19509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:22.038975 19509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:22.039188 19509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:22.039481 19509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:22.039845 19509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:22.039906 19509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.039950 19509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:22.039978 19509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.046754 19509 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.193:33033
I20260620 08:08:22.046820 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.193:33033 every 8 connection(s)
I20260620 08:08:22.047185 19509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/data/info.pb
I20260620 08:08:22.050559 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19509
I20260620 08:08:22.050680 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-0/wal/instance
I20260620 08:08:22.050994 32207 external_mini_cluster.cc:1428] Setting key fb5ae2e7eac266eb428c4a0b3a9b5750
I20260620 08:08:22.052435 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.194:0
--local_ip_for_outbound_sockets=127.31.115.194
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:45319
--builtin_ntp_servers=127.31.115.212:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:08:22.054853 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.048298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:59297 (local address 127.31.115.254:45319)
0620 08:08:22.048349 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.048352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.048589 (+ 237us) server_negotiation.cc:407] Connection header received
0620 08:08:22.049086 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.049087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.049099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.049168 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:22.050352 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.051301 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.052775 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.053121 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.053206 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.054411 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:22.054413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:22.054413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:22.054420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:22.054440 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:22.054445 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:22.054518 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.054693 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.054736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:08:22.055043 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.048176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:45319 (local address 127.31.115.193:59297)
0620 08:08:22.048402 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.048448 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.048947 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.049376 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.049380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.049390 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:22.050119 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.050126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.051442 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.051443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.052672 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.052676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.053268 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.053269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.053512 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.054235 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:22.054235 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:22.054269 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:22.054298 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:22.054576 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:22.054584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:22.054587 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.054677 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.054782 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260620 08:08:22.055649 19637 heartbeater.cc:347] Connected to a master server at 127.31.115.254:45319
I20260620 08:08:22.055794 19637 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:22.056147 19637 heartbeater.cc:511] Master 127.31.115.254:45319 requested a full tablet report, sending...
I20260620 08:08:22.056958 19447 ts_manager.cc:194] Registered new tserver with Master: ba1e1057b0e54f078b62ac87486c6262 (127.31.115.193:33033)
I20260620 08:08:22.058306 19447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.193:59297
W20260620 08:08:22.155019 19640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:22.155381 19640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:22.155427 19640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:22.157784 19640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:22.157862 19640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:22.157913 19640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.194
I20260620 08:08:22.161055 19640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.115.194
--webserver_port=0
--tserver_master_addrs=127.31.115.254:45319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.194
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:22.161396 19640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:22.161860 19640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:22.168056 19648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:22.168056 19645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:22.168048 19646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:22.168322 19640 server_base.cc:1034] running on GCE node
I20260620 08:08:22.168510 19640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:22.168808 19640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:22.169982 19640 hybrid_clock.cc:648] HybridClock initialized: now 1781942902169943 us; error 48 us; skew 500 ppm
I20260620 08:08:22.171484 19640 webserver.cc:460] Webserver started at http://127.31.115.194:41995/ using document root <none> and password file <none>
I20260620 08:08:22.171746 19640 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:22.171823 19640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:22.171975 19640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:22.173113 19640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data/instance:
uuid: "30cd2ba5d20c4077a7fd1878d1c295bb"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "67565c9496d937d0d49336aab8c1af22"
server_key_iv: "2fd9c553c62b0ef93339460dd822e64d"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.173506 19640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance:
uuid: "30cd2ba5d20c4077a7fd1878d1c295bb"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "67565c9496d937d0d49336aab8c1af22"
server_key_iv: "2fd9c553c62b0ef93339460dd822e64d"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.175333 19640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 08:08:22.176265 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.176548 19640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:22.176868 19640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal
uuid: "30cd2ba5d20c4077a7fd1878d1c295bb"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "67565c9496d937d0d49336aab8c1af22"
server_key_iv: "2fd9c553c62b0ef93339460dd822e64d"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.176970 19640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:22.211344 19640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:22.211946 19640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:22.212334 19640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:22.212693 19640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:22.213352 19640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:22.213407 19640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.213455 19640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:22.213490 19640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.221841 19640 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.194:40873
I20260620 08:08:22.221889 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.194:40873 every 8 connection(s)
I20260620 08:08:22.222249 19640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/data/info.pb
I20260620 08:08:22.228837 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.223108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.194:41163 (local address 127.31.115.254:45319)
0620 08:08:22.223197 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.223199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.223638 (+ 439us) server_negotiation.cc:407] Connection header received
0620 08:08:22.224606 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.224608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.224623 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.224682 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:22.225433 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.226074 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.227143 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.227359 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.227452 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.228441 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:22.228443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:22.228444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:22.228450 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:22.228470 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:22.228476 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:22.228566 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.228697 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.228728 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:08:22.229034 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.223096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:45319 (local address 127.31.115.194:41163)
0620 08:08:22.223328 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.223368 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.224434 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.224767 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.224771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.224783 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:22.225308 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.225321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.226199 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.226200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.227025 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.227027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.227467 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.227468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.227609 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.228246 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:22.228247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:22.228277 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:22.228303 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:22.228591 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:22.228598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:22.228601 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.228678 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.228769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260620 08:08:22.229566 19768 heartbeater.cc:347] Connected to a master server at 127.31.115.254:45319
I20260620 08:08:22.229704 19768 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:22.229935 19768 heartbeater.cc:511] Master 127.31.115.254:45319 requested a full tablet report, sending...
I20260620 08:08:22.230502 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19640
I20260620 08:08:22.230590 19447 ts_manager.cc:194] Registered new tserver with Master: 30cd2ba5d20c4077a7fd1878d1c295bb (127.31.115.194:40873)
I20260620 08:08:22.230629 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal/instance
I20260620 08:08:22.230865 32207 external_mini_cluster.cc:1428] Setting key 4d7c76bebcf31dfafeb91c8092eb8508
I20260620 08:08:22.231103 19447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.194:41163
I20260620 08:08:22.232038 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.115.195:0
--local_ip_for_outbound_sockets=127.31.115.195
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:45319
--builtin_ntp_servers=127.31.115.212:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:08:22.326115 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:22.326421 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:22.326447 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:22.327776 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:22.327915 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:22.328219 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.115.195
I20260620 08:08:22.329885 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.115.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.115.195
--webserver_port=0
--tserver_master_addrs=127.31.115.254:45319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.115.195
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:22.330113 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:22.330401 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:22.336238 19776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:22.336321 19777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:22.336345 19771 server_base.cc:1034] running on GCE node
W20260620 08:08:22.336238 19779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:22.336697 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:22.336930 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:22.338121 19771 hybrid_clock.cc:648] HybridClock initialized: now 1781942902338096 us; error 44 us; skew 500 ppm
I20260620 08:08:22.339684 19771 webserver.cc:460] Webserver started at http://127.31.115.195:33263/ using document root <none> and password file <none>
I20260620 08:08:22.339994 19771 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:22.340049 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:22.340197 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:22.341427 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data/instance:
uuid: "8cd76088e4cb43d9bb6f582533c22b84"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "53b4a0af758a3f18a4c024403499569d"
server_key_iv: "836db9945f76d7bae3f7cd94bc99106b"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.341858 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance:
uuid: "8cd76088e4cb43d9bb6f582533c22b84"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "53b4a0af758a3f18a4c024403499569d"
server_key_iv: "836db9945f76d7bae3f7cd94bc99106b"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.343349 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:22.344328 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.344622 19771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:22.344763 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal
uuid: "8cd76088e4cb43d9bb6f582533c22b84"
format_stamp: "Formatted at 2026-06-20 08:08:22 on dist-test-slave-c4w8"
server_key: "53b4a0af758a3f18a4c024403499569d"
server_key_iv: "836db9945f76d7bae3f7cd94bc99106b"
server_key_version: "encryptionkey@0"
I20260620 08:08:22.344856 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:22.373507 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:22.373983 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:22.374172 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:22.374464 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:08:22.375161 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:08:22.375236 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.375275 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:08:22.375330 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:22.383368 19771 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.195:40515
I20260620 08:08:22.383427 19898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.195:40515 every 8 connection(s)
I20260620 08:08:22.383828 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/data/info.pb
I20260620 08:08:22.390491 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.384701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:45319 (local address 127.31.115.195:40825)
0620 08:08:22.384912 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.384954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.385454 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.385933 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.385937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.385946 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:22.386498 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.386502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.387290 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.387291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.388147 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.388151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.388606 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.388608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.388768 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.389586 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:08:22.389587 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:22.389714 (+ 127us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:08:22.389751 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:22.390052 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:22.390059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:22.390063 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.390143 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.390279 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260620 08:08:22.390501 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.384826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.195:40825 (local address 127.31.115.254:45319)
0620 08:08:22.384882 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.384885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.385119 (+ 234us) server_negotiation.cc:407] Connection header received
0620 08:08:22.385723 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.385725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.385741 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.385837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:22.386642 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.387157 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.388273 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.388474 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.388546 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.389892 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:22.389894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:22.389900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:22.389907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:22.389923 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:22.389928 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:22.389995 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.390344 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.390380 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:08:22.390702 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19771
I20260620 08:08:22.390822 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal/instance
I20260620 08:08:22.391041 32207 external_mini_cluster.cc:1428] Setting key 799e8a855fa015328eea0e6a1eb37cb7
I20260620 08:08:22.391498 19899 heartbeater.cc:347] Connected to a master server at 127.31.115.254:45319
I20260620 08:08:22.391611 19899 heartbeater.cc:464] Registering TS with master...
I20260620 08:08:22.391842 19899 heartbeater.cc:511] Master 127.31.115.254:45319 requested a full tablet report, sending...
I20260620 08:08:22.392302 19447 ts_manager.cc:194] Registered new tserver with Master: 8cd76088e4cb43d9bb6f582533c22b84 (127.31.115.195:40515)
I20260620 08:08:22.392714 19447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.115.195:40825
I20260620 08:08:22.403919 32207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:08:22.409368 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.405665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.31.115.254:45319)
0620 08:08:22.405733 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.405735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.405968 (+ 233us) server_negotiation.cc:407] Connection header received
0620 08:08:22.405998 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.405999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.406010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.406077 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:22.406444 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.406999 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.407849 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.408046 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.408134 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.408980 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:22.408982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:22.408982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:22.408987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:22.409003 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:22.409007 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:22.409070 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.409268 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.409294 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:08:22.412521 19447 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:08:22.415298 19447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:08:22.423754 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.418882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33033 (local address 127.0.0.1:50020)
0620 08:08:22.419352 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.419396 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.419478 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.419851 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.419860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.419862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:22.419961 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.419963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.421391 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.421392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.422654 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.422657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.423404 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.423405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.423516 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.423533 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.423602 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.423661 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260620 08:08:22.424189 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.418958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.31.115.193:33033)
0620 08:08:22.419184 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.419189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.419569 (+ 380us) server_negotiation.cc:407] Connection header received
0620 08:08:22.419657 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.419660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.419677 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.419761 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:22.420048 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.421285 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.422762 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.423299 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.423405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.423508 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.424051 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.424097 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260620 08:08:22.424991 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.418944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.31.115.195:40515)
0620 08:08:22.420136 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.420140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.420149 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:08:22.420191 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.420193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.420207 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.420297 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:22.420471 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.421974 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.423826 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.424530 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.424598 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.424679 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.424867 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.424906 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":144,"threads_started":1}
I20260620 08:08:22.425045 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.418846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:40515 (local address 127.0.0.1:41860)
0620 08:08:22.419354 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.419396 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.419478 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.420296 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.420297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.420300 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:22.420352 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.420356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.422223 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.422224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.423727 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.423730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.424709 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.424710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.424757 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.424760 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.424848 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.424900 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260620 08:08:22.425168 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.419090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:40873 (local address 127.0.0.1:44278)
0620 08:08:22.420383 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.420394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.420442 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.420648 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.420649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.420650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:22.420689 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.420691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.421787 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.421790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.423492 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.423496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.424742 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.424743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.424804 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.424808 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.424861 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.424915 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260620 08:08:22.425050 19571 tablet_service.cc:1467] Processing CreateTablet for tablet c69a5b226b2440019b4f3be8572f346f (DEFAULT_TABLE table=test-table [id=6d80b17592304f5aa7e47f43924cd569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:22.425504 19571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69a5b226b2440019b4f3be8572f346f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:22.425546 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.419490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.31.115.194:40873)
0620 08:08:22.419840 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.419844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.420505 (+ 661us) server_negotiation.cc:407] Connection header received
0620 08:08:22.420550 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.420552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.420564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.420651 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:22.420748 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.421543 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.423578 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.424540 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.424612 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.424692 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.425406 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.425462 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":117,"threads_started":1}
I20260620 08:08:22.425750 19833 tablet_service.cc:1467] Processing CreateTablet for tablet c69a5b226b2440019b4f3be8572f346f (DEFAULT_TABLE table=test-table [id=6d80b17592304f5aa7e47f43924cd569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:22.426097 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69a5b226b2440019b4f3be8572f346f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:22.426296 19702 tablet_service.cc:1467] Processing CreateTablet for tablet c69a5b226b2440019b4f3be8572f346f (DEFAULT_TABLE table=test-table [id=6d80b17592304f5aa7e47f43924cd569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:08:22.426647 19702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c69a5b226b2440019b4f3be8572f346f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:22.429330 19916 tablet_bootstrap.cc:492] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Bootstrap starting.
I20260620 08:08:22.430150 19916 tablet_bootstrap.cc:654] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:22.430167 19917 tablet_bootstrap.cc:492] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: Bootstrap starting.
I20260620 08:08:22.430339 19918 tablet_bootstrap.cc:492] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Bootstrap starting.
I20260620 08:08:22.430725 19916 log.cc:824] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:22.430889 19917 tablet_bootstrap.cc:654] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:22.431044 19918 tablet_bootstrap.cc:654] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:22.431213 19917 log.cc:824] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:22.431427 19918 log.cc:824] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:22.431735 19916 tablet_bootstrap.cc:492] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: No bootstrap required, opened a new log
I20260620 08:08:22.431798 19916 ts_tablet_manager.cc:1397] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:08:22.432145 19917 tablet_bootstrap.cc:492] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: No bootstrap required, opened a new log
I20260620 08:08:22.432206 19917 ts_tablet_manager.cc:1397] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:08:22.434310 19918 tablet_bootstrap.cc:492] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: No bootstrap required, opened a new log
I20260620 08:08:22.434396 19918 ts_tablet_manager.cc:1397] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 08:08:22.434469 19916 raft_consensus.cc:348] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.434648 19916 raft_consensus.cc:374] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:22.434684 19916 raft_consensus.cc:729] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd76088e4cb43d9bb6f582533c22b84, State: Initialized, Role: FOLLOWER
I20260620 08:08:22.434957 19916 consensus_queue.cc:260] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.435127 19917 raft_consensus.cc:348] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.435293 19916 ts_tablet_manager.cc:1428] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 08:08:22.435316 19917 raft_consensus.cc:374] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:22.435359 19917 raft_consensus.cc:729] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1e1057b0e54f078b62ac87486c6262, State: Initialized, Role: FOLLOWER
I20260620 08:08:22.435428 19899 heartbeater.cc:503] Master 127.31.115.254:45319 was elected leader, sending a full tablet report...
I20260620 08:08:22.435549 19917 consensus_queue.cc:260] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.435848 19917 ts_tablet_manager.cc:1428] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:08:22.436087 19637 heartbeater.cc:503] Master 127.31.115.254:45319 was elected leader, sending a full tablet report...
I20260620 08:08:22.438153 19918 raft_consensus.cc:348] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.438349 19918 raft_consensus.cc:374] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:22.438408 19918 raft_consensus.cc:729] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cd2ba5d20c4077a7fd1878d1c295bb, State: Initialized, Role: FOLLOWER
I20260620 08:08:22.438552 19918 consensus_queue.cc:260] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.439013 19918 ts_tablet_manager.cc:1428] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:08:22.439025 19768 heartbeater.cc:503] Master 127.31.115.254:45319 was elected leader, sending a full tablet report...
W20260620 08:08:22.472934 19769 tablet.cc:2369] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:08:22.548211 19638 tablet.cc:2369] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:08:22.634516 19900 tablet.cc:2369] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:08:22.643836 19923 raft_consensus.cc:482] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:08:22.643986 19923 raft_consensus.cc:504] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.644479 19923 leader_election.cc:288] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cd76088e4cb43d9bb6f582533c22b84 (127.31.115.195:40515), 30cd2ba5d20c4077a7fd1878d1c295bb (127.31.115.194:40873)
I20260620 08:08:22.649755 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.644909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:40389 (local address 127.31.115.195:40515)
0620 08:08:22.645000 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.645002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.645016 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:08:22.645073 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.645074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.645085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.645339 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:22.645599 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.646449 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.648582 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.649176 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.649270 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.649280 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.649594 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.649682 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:08:22.649762 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.644762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.195:40515 (local address 127.31.115.193:40389)
0620 08:08:22.644914 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.644923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.644999 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.645376 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.645377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.645381 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:22.645465 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.645468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.646773 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.646775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.648426 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.648430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.649362 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.649363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.649462 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.649470 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.649549 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.649657 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":99,"thread_start_us":97,"threads_started":1}
I20260620 08:08:22.649801 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.644800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.115.193:41339 (local address 127.31.115.194:40873)
0620 08:08:22.644848 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.644851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.645018 (+ 167us) server_negotiation.cc:407] Connection header received
0620 08:08:22.645065 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.645066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.645083 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.645368 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:08:22.645598 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.646471 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.648582 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.649212 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.649280 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.649288 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.649678 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.649725 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:08:22.649761 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.644737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.194:40873 (local address 127.31.115.193:41339)
0620 08:08:22.644905 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.644917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:22.645002 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:22.645392 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:22.645393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:22.645394 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:08:22.645466 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.645468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.646782 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.646783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.648426 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:22.648430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.649361 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.649363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:22.649462 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.649471 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 08:08:22.649549 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 08:08:22.649637 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":138,"thread_start_us":98,"threads_started":1}
I20260620 08:08:22.650472 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a5b226b2440019b4f3be8572f346f" candidate_uuid: "ba1e1057b0e54f078b62ac87486c6262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" is_pre_election: true
I20260620 08:08:22.650318 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a5b226b2440019b4f3be8572f346f" candidate_uuid: "ba1e1057b0e54f078b62ac87486c6262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd76088e4cb43d9bb6f582533c22b84" is_pre_election: true
I20260620 08:08:22.650684 19722 raft_consensus.cc:2454] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1e1057b0e54f078b62ac87486c6262 in term 0.
I20260620 08:08:22.650687 19853 raft_consensus.cc:2454] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1e1057b0e54f078b62ac87486c6262 in term 0.
I20260620 08:08:22.651000 19526 leader_election.cc:302] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cd2ba5d20c4077a7fd1878d1c295bb, ba1e1057b0e54f078b62ac87486c6262; no voters:
I20260620 08:08:22.651172 19923 raft_consensus.cc:2781] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:08:22.651234 19923 raft_consensus.cc:482] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:08:22.651262 19923 raft_consensus.cc:3037] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:22.651979 19923 raft_consensus.cc:504] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.652144 19923 leader_election.cc:288] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [CANDIDATE]: Term 1 election: Requested vote from peers 8cd76088e4cb43d9bb6f582533c22b84 (127.31.115.195:40515), 30cd2ba5d20c4077a7fd1878d1c295bb (127.31.115.194:40873)
I20260620 08:08:22.652325 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a5b226b2440019b4f3be8572f346f" candidate_uuid: "ba1e1057b0e54f078b62ac87486c6262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb"
I20260620 08:08:22.652352 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69a5b226b2440019b4f3be8572f346f" candidate_uuid: "ba1e1057b0e54f078b62ac87486c6262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd76088e4cb43d9bb6f582533c22b84"
I20260620 08:08:22.652441 19722 raft_consensus.cc:3037] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:22.652463 19853 raft_consensus.cc:3037] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:22.653329 19853 raft_consensus.cc:2454] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba1e1057b0e54f078b62ac87486c6262 in term 1.
I20260620 08:08:22.653752 19525 leader_election.cc:302] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cd76088e4cb43d9bb6f582533c22b84, ba1e1057b0e54f078b62ac87486c6262; no voters:
I20260620 08:08:22.653868 19923 raft_consensus.cc:2781] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:22.653833 19722 raft_consensus.cc:2454] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba1e1057b0e54f078b62ac87486c6262 in term 1.
I20260620 08:08:22.654057 19923 raft_consensus.cc:686] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [term 1 LEADER]: Becoming Leader. State: Replica: ba1e1057b0e54f078b62ac87486c6262, State: Running, Role: LEADER
I20260620 08:08:22.654171 19923 consensus_queue.cc:237] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } }
I20260620 08:08:22.655349 19449 catalog_manager.cc:5495] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba1e1057b0e54f078b62ac87486c6262 (127.31.115.193). New cstate: current_term: 1 leader_uuid: "ba1e1057b0e54f078b62ac87486c6262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1e1057b0e54f078b62ac87486c6262" member_type: VOTER last_known_addr { host: "127.31.115.193" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260620 08:08:22.715674 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.712473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.31.115.193:33033)
0620 08:08:22.712599 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:22.712602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:22.712613 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:22.712654 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:22.712656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:22.712683 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:22.712807 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:22.713173 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.713707 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.714655 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:22.714867 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:22.714963 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:22.715282 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:22.715406 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:22.715522 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:22.715548 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 08:08:22.715572 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 08:08:22.718221 19853 raft_consensus.cc:1261] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 1 FOLLOWER]: Refusing update from remote peer ba1e1057b0e54f078b62ac87486c6262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:22.718268 19722 raft_consensus.cc:1261] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 1 FOLLOWER]: Refusing update from remote peer ba1e1057b0e54f078b62ac87486c6262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:22.718763 19923 consensus_queue.cc:1035] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd76088e4cb43d9bb6f582533c22b84" member_type: VOTER last_known_addr { host: "127.31.115.195" port: 40515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:22.718962 19927 consensus_queue.cc:1035] T c69a5b226b2440019b4f3be8572f346f P ba1e1057b0e54f078b62ac87486c6262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cd2ba5d20c4077a7fd1878d1c295bb" member_type: VOTER last_known_addr { host: "127.31.115.194" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:22.722347 19932 mvcc.cc:205] Tried to move back new op lower bound from 7298838129529057280 to 7298838129272774656. Current Snapshot: MvccSnapshot[applied={T|T < 7298838129529057280}]
I20260620 08:08:22.723711 19931 mvcc.cc:205] Tried to move back new op lower bound from 7298838129529057280 to 7298838129272774656. Current Snapshot: MvccSnapshot[applied={T|T < 7298838129529057280}]
I20260620 08:08:22.725272 19447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:22.725383 19447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:08:22.726559 19447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d41be4a28884ae0aa5a94be327ae825: Sending DeleteTablet for 3 replicas of tablet c69a5b226b2440019b4f3be8572f346f
I20260620 08:08:22.726989 19833 tablet_service.cc:1514] Processing DeleteTablet for tablet c69a5b226b2440019b4f3be8572f346f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:22 UTC) from {username='slave'} at 127.0.0.1:41860
I20260620 08:08:22.727097 19702 tablet_service.cc:1514] Processing DeleteTablet for tablet c69a5b226b2440019b4f3be8572f346f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:22 UTC) from {username='slave'} at 127.0.0.1:44278
I20260620 08:08:22.727409 19941 tablet_replica.cc:330] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: stopping tablet replica
I20260620 08:08:22.727443 19571 tablet_service.cc:1514] Processing DeleteTablet for tablet c69a5b226b2440019b4f3be8572f346f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:08:22 UTC) from {username='slave'} at 127.0.0.1:50020
I20260620 08:08:22.727504 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19509
I20260620 08:08:22.727531 19941 raft_consensus.cc:2229] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:22.728744 19941 raft_consensus.cc:2258] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:22.729303 19942 tablet_replica.cc:330] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: stopping tablet replica
I20260620 08:08:22.729295 19941 ts_tablet_manager.cc:1905] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:22.729436 19942 raft_consensus.cc:2229] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:08:22.729549 19942 raft_consensus.cc:2258] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:08:22.730360 19942 ts_tablet_manager.cc:1905] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:08:22.732008 19941 ts_tablet_manager.cc:1918] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:22.732091 19941 log.cc:1196] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-2/wal/wals/c69a5b226b2440019b4f3be8572f346f
I20260620 08:08:22.732246 19942 ts_tablet_manager.cc:1918] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:08:22.732329 19942 log.cc:1196] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Deleting WAL directory at /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781942645567166-32207-0/minicluster-data/ts-1/wal/wals/c69a5b226b2440019b4f3be8572f346f
I20260620 08:08:22.732396 19941 ts_tablet_manager.cc:1939] T c69a5b226b2440019b4f3be8572f346f P 8cd76088e4cb43d9bb6f582533c22b84: Deleting consensus metadata
I20260620 08:08:22.732731 19942 ts_tablet_manager.cc:1939] T c69a5b226b2440019b4f3be8572f346f P 30cd2ba5d20c4077a7fd1878d1c295bb: Deleting consensus metadata
I20260620 08:08:22.732975 19434 catalog_manager.cc:4841] TS 8cd76088e4cb43d9bb6f582533c22b84 (127.31.115.195:40515): tablet c69a5b226b2440019b4f3be8572f346f (table test-table [id=6d80b17592304f5aa7e47f43924cd569]) successfully deleted
I20260620 08:08:22.733177 19435 catalog_manager.cc:4841] TS 30cd2ba5d20c4077a7fd1878d1c295bb (127.31.115.194:40873): tablet c69a5b226b2440019b4f3be8572f346f (table test-table [id=6d80b17592304f5aa7e47f43924cd569]) successfully deleted
W20260620 08:08:22.734269 19435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.115.193:33033 (error 108)
I20260620 08:08:22.734486 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19640
I20260620 08:08:22.735951 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:22.735763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.193:33033 (local address 127.0.0.1:50046)
0620 08:08:22.735864 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:22.735908 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 08:08:22.736075 19435 catalog_manager.cc:4584] TS ba1e1057b0e54f078b62ac87486c6262 (127.31.115.193:33033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c69a5b226b2440019b4f3be8572f346f: Network error: Client connection negotiation failed: client connection to 127.31.115.193:33033: connect: Connection refused (error 111)
I20260620 08:08:22.739929 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19771
I20260620 08:08:22.745811 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19418
2026-06-20T08:08:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1053 ms)
[----------] 6 tests from EncryptionPolicyTest (9629 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:22 dist-test-slave-c4w8 krb5kdc[19947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:22 dist-test-slave-c4w8 krb5kdc[19947](info): set up 2 sockets
Jun 20 08:08:22 dist-test-slave-c4w8 krb5kdc[19947](info): commencing operation
krb5kdc: starting...
W20260620 08:08:23.859144 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:23 dist-test-slave-c4w8 krb5kdc[19947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:23.998148 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:24.084981 19960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:24.085254 19960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:24.085295 19960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:24.086621 19960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:24.086709 19960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:24.086735 19960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:24.086755 19960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:24.086800 19960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:24.088482 19960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:24.088809 19960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:24.089071 19960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:24.094669 19965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:24.094658 19966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:24.094659 19968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:24.094986 19960 server_base.cc:1034] running on GCE node
I20260620 08:08:24.095381 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:24.095716 19960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:24.096931 19960 hybrid_clock.cc:648] HybridClock initialized: now 1781942904096920 us; error 38 us; skew 500 ppm
Jun 20 08:08:24 dist-test-slave-c4w8 krb5kdc[19947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:24.099915 19960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:24.100531 19960 webserver.cc:460] Webserver started at http://127.0.0.1:34701/ using document root <none> and password file <none>
I20260620 08:08:24.100827 19960 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:24.100955 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:24.101137 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:24.102433 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "586b4e8640314208879b99e5cc12f38a"
format_stamp: "Formatted at 2026-06-20 08:08:24 on dist-test-slave-c4w8"
server_key: "6f438dc9d1e20c9f4324e9de274e6939"
server_key_iv: "6022b4775a670df2469ed37136094a23"
server_key_version: "encryptionkey@0"
I20260620 08:08:24.102897 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "586b4e8640314208879b99e5cc12f38a"
format_stamp: "Formatted at 2026-06-20 08:08:24 on dist-test-slave-c4w8"
server_key: "6f438dc9d1e20c9f4324e9de274e6939"
server_key_iv: "6022b4775a670df2469ed37136094a23"
server_key_version: "encryptionkey@0"
I20260620 08:08:24.104568 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:24.106091 19975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:24.106472 19960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 08:08:24.106604 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "586b4e8640314208879b99e5cc12f38a"
format_stamp: "Formatted at 2026-06-20 08:08:24 on dist-test-slave-c4w8"
server_key: "6f438dc9d1e20c9f4324e9de274e6939"
server_key_iv: "6022b4775a670df2469ed37136094a23"
server_key_version: "encryptionkey@0"
I20260620 08:08:24.106695 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:24.141845 19960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:24.144698 19960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:24.144894 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:24.149406 19960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42037
I20260620 08:08:24.149484 20027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42037 every 8 connection(s)
I20260620 08:08:24.149892 19960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:24.150873 20028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:24.153985 20028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a: Bootstrap starting.
I20260620 08:08:24.154671 20028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:24.154969 20028 log.cc:824] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:24.155391 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 19960
I20260620 08:08:24.155529 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:24.155741 32207 external_mini_cluster.cc:1428] Setting key 4569a7e3fbc826b5690ec3f40d644313
I20260620 08:08:24.155915 20028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a: No bootstrap required, opened a new log
I20260620 08:08:24.157941 20028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b4e8640314208879b99e5cc12f38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260620 08:08:24.158164 20028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:24.158226 20028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 586b4e8640314208879b99e5cc12f38a, State: Initialized, Role: FOLLOWER
I20260620 08:08:24.158390 20028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b4e8640314208879b99e5cc12f38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260620 08:08:24.158489 20028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:24.158524 20028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:24.158571 20028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:24.159878 20028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b4e8640314208879b99e5cc12f38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260620 08:08:24.159997 20028 leader_election.cc:302] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 586b4e8640314208879b99e5cc12f38a; no voters:
I20260620 08:08:24.160171 20028 leader_election.cc:288] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:24.160266 20033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:24.160423 20033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [term 1 LEADER]: Becoming Leader. State: Replica: 586b4e8640314208879b99e5cc12f38a, State: Running, Role: LEADER
I20260620 08:08:24.160431 20028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:24.160545 20033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b4e8640314208879b99e5cc12f38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260620 08:08:24.161286 20034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "586b4e8640314208879b99e5cc12f38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b4e8640314208879b99e5cc12f38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260620 08:08:24.161465 20034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:24.162288 20035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 586b4e8640314208879b99e5cc12f38a. Latest consensus state: current_term: 1 leader_uuid: "586b4e8640314208879b99e5cc12f38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "586b4e8640314208879b99e5cc12f38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260620 08:08:24.162371 20035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a [sys.catalog]: This master's current role is: LEADER
W20260620 08:08:24.162863 20049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:24.162923 20049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:24.163252 20042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:24.163560 20042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:08:24 dist-test-slave-c4w8 krb5kdc[19947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:08:24.165721 20042 catalog_manager.cc:1347] Generated new cluster ID: d0c2d2608ec148549b820b0f1f0a293b
I20260620 08:08:24.165776 20042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:24.170949 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:24.156321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57831 (local address 127.0.0.1:42037)
0620 08:08:24.156534 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:24.156545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:24.158029 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 08:08:24.158262 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:24.158270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:24.158333 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:24.158494 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:24.159267 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:24.160777 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:24.161980 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:24.162719 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:24.162845 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:24.166164 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:24.166169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:24.166169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:24.166183 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:24.168806 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:24.169434 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:24.169438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:24.169438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:24.169589 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:24.169896 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:24.169897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:24.169897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:24.170313 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:24.170487 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:24.170616 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:08:24.170731 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260620 08:08:24.172086 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:24.181900 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:24.173679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39063 (local address 127.0.0.1:42037)
0620 08:08:24.173719 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:24.173721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:24.173844 (+ 123us) server_negotiation.cc:407] Connection header received
0620 08:08:24.173987 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:24.173988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:24.174000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:24.174071 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:24.174754 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:24.175544 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:24.176595 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:24.176933 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:24.176997 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:24.178558 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:24.178560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:24.178560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:24.178568 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:24.180534 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:24.181002 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:24.181003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:24.181003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:24.181064 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:24.181303 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:24.181304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:24.181305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:24.181441 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:24.181534 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:24.181737 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 08:08:24.181800 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:08:24.187875 20042 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:24.188577 20042 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:24.195062 20042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 586b4e8640314208879b99e5cc12f38a: Generated new TSK 0
I20260620 08:08:24.195300 20042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:24.200018 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 19960
W20260620 08:08:24.204900 19945 connection.cc:535] client connection to 127.0.0.1:42037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42037): Connection reset by peer (error 104)
2026-06-20T08:08:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:24 dist-test-slave-c4w8 krb5kdc[20059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:24 dist-test-slave-c4w8 krb5kdc[20059](info): set up 2 sockets
Jun 20 08:08:24 dist-test-slave-c4w8 krb5kdc[20059](info): commencing operation
krb5kdc: starting...
W20260620 08:08:25.294564 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:25 dist-test-slave-c4w8 krb5kdc[20059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:25.430485 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:25.519793 20072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:25.520094 20072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:25.520145 20072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:25.521549 20072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:25.521688 20072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:25.521739 20072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:25.521770 20072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:25.521803 20072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:25.523794 20072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:25.524154 20072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:25.524449 20072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:25.530493 20080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:25.530651 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:25.530473 20077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:25.530849 20072 server_base.cc:1034] running on GCE node
I20260620 08:08:25.531133 20072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:25.531472 20072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:25.532657 20072 hybrid_clock.cc:648] HybridClock initialized: now 1781942905532635 us; error 40 us; skew 500 ppm
Jun 20 08:08:25 dist-test-slave-c4w8 krb5kdc[20059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:25.535686 20072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:25.536199 20072 webserver.cc:460] Webserver started at http://127.0.0.1:41173/ using document root <none> and password file <none>
I20260620 08:08:25.536473 20072 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:25.536556 20072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:25.536736 20072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:25.538107 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "f65d589dc72840c9af262590b581bb07"
format_stamp: "Formatted at 2026-06-20 08:08:25 on dist-test-slave-c4w8"
server_key: "21d0c7826f0ba265d7a574f9877a3bd8"
server_key_iv: "9a8127a42475f8e7a148676315824bd5"
server_key_version: "encryptionkey@0"
I20260620 08:08:25.538534 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "f65d589dc72840c9af262590b581bb07"
format_stamp: "Formatted at 2026-06-20 08:08:25 on dist-test-slave-c4w8"
server_key: "21d0c7826f0ba265d7a574f9877a3bd8"
server_key_iv: "9a8127a42475f8e7a148676315824bd5"
server_key_version: "encryptionkey@0"
I20260620 08:08:25.540182 20072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:25.541170 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:25.541555 20072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:25.541728 20072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "f65d589dc72840c9af262590b581bb07"
format_stamp: "Formatted at 2026-06-20 08:08:25 on dist-test-slave-c4w8"
server_key: "21d0c7826f0ba265d7a574f9877a3bd8"
server_key_iv: "9a8127a42475f8e7a148676315824bd5"
server_key_version: "encryptionkey@0"
I20260620 08:08:25.541851 20072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:25.554425 20072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:25.557384 20072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:25.557582 20072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:25.563414 20072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40639
I20260620 08:08:25.563459 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40639 every 8 connection(s)
I20260620 08:08:25.563913 20072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:25.565054 20140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:25.566157 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20072
I20260620 08:08:25.566282 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:25.566509 32207 external_mini_cluster.cc:1428] Setting key 0bfaeda84521884ffd8f5ed3ad5011f2
I20260620 08:08:25.569257 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07: Bootstrap starting.
I20260620 08:08:25.570056 20140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:25.570394 20140 log.cc:824] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:25.571264 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07: No bootstrap required, opened a new log
I20260620 08:08:25.573544 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:25.567435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46621 (local address 127.0.0.1:40639)
0620 08:08:25.567646 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:25.567656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:25.567697 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:08:25.568278 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:25.568287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:25.568385 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:25.568546 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:25.568791 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:25.570170 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:25.571266 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:25.571839 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:25.571946 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:25.572792 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:25.572797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:25.572797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:25.572979 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:25.573011 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:25.573024 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:25.573098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:25.573304 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:08:25.573366 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260620 08:08:25.573938 20140 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d589dc72840c9af262590b581bb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 08:08:25.574127 20140 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:25.574167 20140 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65d589dc72840c9af262590b581bb07, State: Initialized, Role: FOLLOWER
I20260620 08:08:25.574288 20140 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d589dc72840c9af262590b581bb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 08:08:25.574375 20140 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:25.574407 20140 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:25.574465 20140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:25.574834 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:25.575299 20140 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d589dc72840c9af262590b581bb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 08:08:25.575431 20140 leader_election.cc:302] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65d589dc72840c9af262590b581bb07; no voters:
I20260620 08:08:25.575613 20140 leader_election.cc:288] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:25.575702 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:25.575906 20140 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:25.575934 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [term 1 LEADER]: Becoming Leader. State: Replica: f65d589dc72840c9af262590b581bb07, State: Running, Role: LEADER
I20260620 08:08:25.576062 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d589dc72840c9af262590b581bb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 08:08:25.576452 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65d589dc72840c9af262590b581bb07. Latest consensus state: current_term: 1 leader_uuid: "f65d589dc72840c9af262590b581bb07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d589dc72840c9af262590b581bb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260620 08:08:25.576563 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:25.576512 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65d589dc72840c9af262590b581bb07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d589dc72840c9af262590b581bb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260620 08:08:25.576738 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07 [sys.catalog]: This master's current role is: LEADER
W20260620 08:08:25.577921 20165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:25.577999 20165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:25.578051 20158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:25.578212 20158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:25.579704 20158 catalog_manager.cc:1347] Generated new cluster ID: 09a33097242741ca865d6ebba2dfc2ff
I20260620 08:08:25.579756 20158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:25.580832 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:25.576564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45239 (local address 127.0.0.1:40639)
0620 08:08:25.576610 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:25.576612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:25.577076 (+ 464us) server_negotiation.cc:407] Connection header received
0620 08:08:25.577194 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:25.577195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:25.577207 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:25.577237 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:25.577516 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:25.578310 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:25.579345 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:25.579655 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:25.579699 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:25.580556 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:25.580558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:25.580558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:25.580564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:25.580578 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:25.580583 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:25.580615 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:25.580746 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 08:08:25.580758 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:08:25.592622 20158 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:25.593184 20158 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:25.598050 20158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65d589dc72840c9af262590b581bb07: Generated new TSK 0
I20260620 08:08:25.598325 20158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:25.601122 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20072
W20260620 08:08:25.605854 20057 connection.cc:535] client connection to 127.0.0.1:40639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40639): Connection reset by peer (error 104)
2026-06-20T08:08:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:25 dist-test-slave-c4w8 krb5kdc[20171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:25 dist-test-slave-c4w8 krb5kdc[20171](info): set up 2 sockets
Jun 20 08:08:25 dist-test-slave-c4w8 krb5kdc[20171](info): commencing operation
krb5kdc: starting...
W20260620 08:08:26.694926 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:26 dist-test-slave-c4w8 krb5kdc[20171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:26.829226 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:26.913494 20184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:26.913748 20184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:26.913774 20184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:26.915087 20184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:26.915151 20184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:26.915169 20184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:26.915182 20184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:26.915195 20184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:26.916862 20184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:26.917147 20184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:26.917409 20184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:26.922763 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:26.922757 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:26.922763 20192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:26.922895 20184 server_base.cc:1034] running on GCE node
I20260620 08:08:26.923213 20184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:26.923594 20184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:26.924785 20184 hybrid_clock.cc:648] HybridClock initialized: now 1781942906924769 us; error 38 us; skew 500 ppm
Jun 20 08:08:26 dist-test-slave-c4w8 krb5kdc[20171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:26.927773 20184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:26.928447 20184 webserver.cc:460] Webserver started at http://127.0.0.1:39327/ using document root <none> and password file <none>
I20260620 08:08:26.928745 20184 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:26.928821 20184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:26.928969 20184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:26.930552 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "b90a7b6c1d154ec885a450ee949b2d55"
format_stamp: "Formatted at 2026-06-20 08:08:26 on dist-test-slave-c4w8"
server_key: "0a2874d452812aaa6666a1b928da3f67"
server_key_iv: "5a3f49f546a1dcf88723ca701261680b"
server_key_version: "encryptionkey@0"
I20260620 08:08:26.931052 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "b90a7b6c1d154ec885a450ee949b2d55"
format_stamp: "Formatted at 2026-06-20 08:08:26 on dist-test-slave-c4w8"
server_key: "0a2874d452812aaa6666a1b928da3f67"
server_key_iv: "5a3f49f546a1dcf88723ca701261680b"
server_key_version: "encryptionkey@0"
I20260620 08:08:26.932543 20184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:26.933480 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:26.933779 20184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:26.933873 20184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "b90a7b6c1d154ec885a450ee949b2d55"
format_stamp: "Formatted at 2026-06-20 08:08:26 on dist-test-slave-c4w8"
server_key: "0a2874d452812aaa6666a1b928da3f67"
server_key_iv: "5a3f49f546a1dcf88723ca701261680b"
server_key_version: "encryptionkey@0"
I20260620 08:08:26.933954 20184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:26.937870 20184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:26.940244 20184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:26.940367 20184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:26.945084 20184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42857
I20260620 08:08:26.945119 20251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42857 every 8 connection(s)
I20260620 08:08:26.945538 20184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:26.946839 20252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:26.949393 20252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55: Bootstrap starting.
I20260620 08:08:26.950114 20252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:26.950438 20252 log.cc:824] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:26.951301 20252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55: No bootstrap required, opened a new log
I20260620 08:08:26.953123 20252 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a7b6c1d154ec885a450ee949b2d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260620 08:08:26.953275 20252 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:26.953301 20252 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90a7b6c1d154ec885a450ee949b2d55, State: Initialized, Role: FOLLOWER
I20260620 08:08:26.953475 20252 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a7b6c1d154ec885a450ee949b2d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260620 08:08:26.953553 20252 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:26.953640 20252 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:26.953719 20252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:26.954376 20252 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a7b6c1d154ec885a450ee949b2d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260620 08:08:26.954478 20252 leader_election.cc:302] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90a7b6c1d154ec885a450ee949b2d55; no voters:
I20260620 08:08:26.954679 20252 leader_election.cc:288] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:26.954828 20255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:26.954936 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20184
I20260620 08:08:26.954958 20252 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:26.955073 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:26.955073 20255 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [term 1 LEADER]: Becoming Leader. State: Replica: b90a7b6c1d154ec885a450ee949b2d55, State: Running, Role: LEADER
I20260620 08:08:26.955180 20255 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a7b6c1d154ec885a450ee949b2d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260620 08:08:26.955307 32207 external_mini_cluster.cc:1428] Setting key 20025efe78ab00804c4c8b9302f0154d
I20260620 08:08:26.956048 20257 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90a7b6c1d154ec885a450ee949b2d55. Latest consensus state: current_term: 1 leader_uuid: "b90a7b6c1d154ec885a450ee949b2d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a7b6c1d154ec885a450ee949b2d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } } }
I20260620 08:08:26.956151 20256 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90a7b6c1d154ec885a450ee949b2d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a7b6c1d154ec885a450ee949b2d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } } }
I20260620 08:08:26.956213 20256 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:26.956153 20257 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:26.956516 20264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:26.957155 20264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:26.958905 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:26.955866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50467 (local address 127.0.0.1:42857)
0620 08:08:26.956717 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:26.956728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:26.956767 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:08:26.956891 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:26.956911 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:26.956946 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:26.957481 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:26.958233 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:26.958237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:26.958237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:26.958241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:26.958275 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:26.958290 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:26.958364 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:26.958643 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 08:08:26.958688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":88,"threads_started":1}
I20260620 08:08:26.959631 20264 catalog_manager.cc:1347] Generated new cluster ID: 9779c798b77243fd8a76907b22555975
I20260620 08:08:26.959693 20264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:26.960115 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:26.963012 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:26.961767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38995 (local address 127.0.0.1:42857)
0620 08:08:26.961855 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:26.961857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:26.961923 (+ 66us) server_negotiation.cc:407] Connection header received
0620 08:08:26.961997 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:26.961997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:26.962003 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:26.962049 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:26.962713 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:26.962714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:26.962715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:26.962719 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:26.962736 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:26.962741 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:26.962807 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:26.962910 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:08:26.962941 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:08:26.993880 20264 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:26.994122 20264 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:27.003386 20264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90a7b6c1d154ec885a450ee949b2d55: Generated new TSK 0
I20260620 08:08:27.003638 20264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:27.012545 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20184
2026-06-20T08:08:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:27 dist-test-slave-c4w8 krb5kdc[20283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:27 dist-test-slave-c4w8 krb5kdc[20283](info): set up 2 sockets
Jun 20 08:08:27 dist-test-slave-c4w8 krb5kdc[20283](info): commencing operation
krb5kdc: starting...
W20260620 08:08:28.110247 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:28 dist-test-slave-c4w8 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:28.248991 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:28.337846 20296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:28.338102 20296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:28.338142 20296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:28.339371 20296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:28.339448 20296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:28.339474 20296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:28.339509 20296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:28.339537 20296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:28.341178 20296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:28.341480 20296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:28.341774 20296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:28.347170 20302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:28.347198 20304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:28.347227 20296 server_base.cc:1034] running on GCE node
W20260620 08:08:28.347177 20301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:28.347759 20296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:28.348078 20296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:28.349287 20296 hybrid_clock.cc:648] HybridClock initialized: now 1781942908349273 us; error 30 us; skew 500 ppm
Jun 20 08:08:28 dist-test-slave-c4w8 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:28.352041 20296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:28.352559 20296 webserver.cc:460] Webserver started at http://127.0.0.1:44041/ using document root <none> and password file <none>
I20260620 08:08:28.352846 20296 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:28.352926 20296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:28.353084 20296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:28.354365 20296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "09f911dc669346b8ad7b50d86d17b635"
format_stamp: "Formatted at 2026-06-20 08:08:28 on dist-test-slave-c4w8"
server_key: "34d579e2864cd68d5325f32e97f504b4"
server_key_iv: "e083bef9523f3c4dbb8d6f62cacb4109"
server_key_version: "encryptionkey@0"
I20260620 08:08:28.354759 20296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "09f911dc669346b8ad7b50d86d17b635"
format_stamp: "Formatted at 2026-06-20 08:08:28 on dist-test-slave-c4w8"
server_key: "34d579e2864cd68d5325f32e97f504b4"
server_key_iv: "e083bef9523f3c4dbb8d6f62cacb4109"
server_key_version: "encryptionkey@0"
I20260620 08:08:28.356235 20296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:28.357056 20311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:28.357306 20296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:28.357432 20296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "09f911dc669346b8ad7b50d86d17b635"
format_stamp: "Formatted at 2026-06-20 08:08:28 on dist-test-slave-c4w8"
server_key: "34d579e2864cd68d5325f32e97f504b4"
server_key_iv: "e083bef9523f3c4dbb8d6f62cacb4109"
server_key_version: "encryptionkey@0"
I20260620 08:08:28.357511 20296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:28.375344 20296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:28.378017 20296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:28.378204 20296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:28.382551 20296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33777
I20260620 08:08:28.382597 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33777 every 8 connection(s)
I20260620 08:08:28.382993 20296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:28.383965 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:28.384611 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20296
I20260620 08:08:28.384728 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:28.384958 32207 external_mini_cluster.cc:1428] Setting key 1eff53c8ac66fca7790fd904bddf2e9e
I20260620 08:08:28.387116 20364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635: Bootstrap starting.
I20260620 08:08:28.387737 20364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:28.388048 20364 log.cc:824] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:28.388751 20364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635: No bootstrap required, opened a new log
I20260620 08:08:28.390882 20364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f911dc669346b8ad7b50d86d17b635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 08:08:28.391048 20364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:28.391099 20364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f911dc669346b8ad7b50d86d17b635, State: Initialized, Role: FOLLOWER
I20260620 08:08:28.391260 20364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f911dc669346b8ad7b50d86d17b635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 08:08:28.391361 20364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:28.391413 20364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:28.391474 20364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:28.392231 20364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f911dc669346b8ad7b50d86d17b635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 08:08:28.392345 20364 leader_election.cc:302] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f911dc669346b8ad7b50d86d17b635; no voters:
Jun 20 08:08:28 dist-test-slave-c4w8 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:08:28.392527 20364 leader_election.cc:288] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:28.392575 20369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:28.392778 20369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [term 1 LEADER]: Becoming Leader. State: Replica: 09f911dc669346b8ad7b50d86d17b635, State: Running, Role: LEADER
I20260620 08:08:28.392824 20364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:28.392949 20369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f911dc669346b8ad7b50d86d17b635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 08:08:28.393352 20370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f911dc669346b8ad7b50d86d17b635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f911dc669346b8ad7b50d86d17b635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } } }
I20260620 08:08:28.393457 20370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:28.393728 20371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f911dc669346b8ad7b50d86d17b635. Latest consensus state: current_term: 1 leader_uuid: "09f911dc669346b8ad7b50d86d17b635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f911dc669346b8ad7b50d86d17b635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } } }
I20260620 08:08:28.393847 20371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:28.394165 20376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:28.395710 20376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:28.396914 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:28.385541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55723 (local address 127.0.0.1:33777)
0620 08:08:28.385909 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:28.385918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:28.385946 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:08:28.386387 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:28.386393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:28.386442 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:28.386606 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:28.387352 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:28.388405 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:28.389303 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:28.389817 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:28.389958 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:28.393044 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:28.393048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:28.393048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:28.393062 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:28.395179 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:28.395613 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:28.395617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:28.395617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:28.395719 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:28.395960 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:28.395961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:28.395961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:28.396214 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:28.396455 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:28.396525 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 08:08:28.396675 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":103,"threads_started":1}
I20260620 08:08:28.397388 20376 catalog_manager.cc:1347] Generated new cluster ID: 1a3fdade112a41efafe3dc1c26890140
I20260620 08:08:28.397437 20376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:28.397954 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:28.403126 20376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:28.403949 20376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:28.407372 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:28.399546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34373 (local address 127.0.0.1:33777)
0620 08:08:28.399608 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:28.399610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:28.399846 (+ 236us) server_negotiation.cc:407] Connection header received
0620 08:08:28.399910 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:28.399911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:28.399921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:28.399975 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:28.400654 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:28.401410 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:28.402584 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:28.402891 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:28.402985 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:28.404531 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:28.404532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:28.404532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:28.404540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:28.406147 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:28.406544 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:28.406545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:28.406545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:28.406603 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:28.406829 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:28.406830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:28.406830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:28.406950 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:28.407038 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:28.407184 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:08:28.407281 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:08:28.416069 20376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f911dc669346b8ad7b50d86d17b635: Generated new TSK 0
I20260620 08:08:28.416283 20376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:28.424044 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20296
2026-06-20T08:08:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:28 dist-test-slave-c4w8 krb5kdc[20395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:28 dist-test-slave-c4w8 krb5kdc[20395](info): set up 2 sockets
Jun 20 08:08:28 dist-test-slave-c4w8 krb5kdc[20395](info): commencing operation
krb5kdc: starting...
W20260620 08:08:29.500006 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:29 dist-test-slave-c4w8 krb5kdc[20395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:29.633140 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:29.722864 20408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:29.723181 20408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:29.723225 20408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:29.724488 20408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:29.724576 20408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:29.724601 20408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:29.724625 20408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:29.724654 20408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:29.726459 20408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:29.726778 20408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:29.727036 20408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:29.732901 20414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:29.732885 20413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:29.732923 20416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:29.733232 20408 server_base.cc:1034] running on GCE node
I20260620 08:08:29.733542 20408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:29.733876 20408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:29.735070 20408 hybrid_clock.cc:648] HybridClock initialized: now 1781942909735056 us; error 44 us; skew 500 ppm
Jun 20 08:08:29 dist-test-slave-c4w8 krb5kdc[20395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:29.737775 20408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:29.738273 20408 webserver.cc:460] Webserver started at http://127.0.0.1:33613/ using document root <none> and password file <none>
I20260620 08:08:29.738471 20408 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:29.738512 20408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:29.738668 20408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:29.739745 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "69fba2a84243490593155d09870b7162"
format_stamp: "Formatted at 2026-06-20 08:08:29 on dist-test-slave-c4w8"
server_key: "7e861b16ecb7aa1a2a7759a80ac7ff68"
server_key_iv: "7ea12f5b01f009af13a86e839048ad28"
server_key_version: "encryptionkey@0"
I20260620 08:08:29.740062 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "69fba2a84243490593155d09870b7162"
format_stamp: "Formatted at 2026-06-20 08:08:29 on dist-test-slave-c4w8"
server_key: "7e861b16ecb7aa1a2a7759a80ac7ff68"
server_key_iv: "7ea12f5b01f009af13a86e839048ad28"
server_key_version: "encryptionkey@0"
I20260620 08:08:29.741400 20408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:29.742229 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:29.742419 20408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:08:29.742515 20408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "69fba2a84243490593155d09870b7162"
format_stamp: "Formatted at 2026-06-20 08:08:29 on dist-test-slave-c4w8"
server_key: "7e861b16ecb7aa1a2a7759a80ac7ff68"
server_key_iv: "7ea12f5b01f009af13a86e839048ad28"
server_key_version: "encryptionkey@0"
I20260620 08:08:29.742592 20408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:29.755347 20408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:29.758072 20408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:29.758227 20408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:29.763021 20408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41367
I20260620 08:08:29.763044 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41367 every 8 connection(s)
I20260620 08:08:29.763458 20408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:29.764423 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:29.766927 20476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162: Bootstrap starting.
I20260620 08:08:29.767650 20476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:29.767956 20476 log.cc:824] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:29.768873 20476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162: No bootstrap required, opened a new log
I20260620 08:08:29.769199 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20408
I20260620 08:08:29.769383 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:29.769634 32207 external_mini_cluster.cc:1428] Setting key 54ac313cc69d8030005d738220edd542
I20260620 08:08:29.771132 20476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69fba2a84243490593155d09870b7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41367 } }
I20260620 08:08:29.771306 20476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:29.771369 20476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fba2a84243490593155d09870b7162, State: Initialized, Role: FOLLOWER
I20260620 08:08:29.771538 20476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69fba2a84243490593155d09870b7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41367 } }
I20260620 08:08:29.771638 20476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:29.771684 20476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:29.771749 20476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:29.772359 20476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69fba2a84243490593155d09870b7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41367 } }
I20260620 08:08:29.772487 20476 leader_election.cc:302] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69fba2a84243490593155d09870b7162; no voters:
I20260620 08:08:29.772696 20476 leader_election.cc:288] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:29.772758 20481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:29.772919 20481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [term 1 LEADER]: Becoming Leader. State: Replica: 69fba2a84243490593155d09870b7162, State: Running, Role: LEADER
I20260620 08:08:29.772989 20476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:29.774641 20481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69fba2a84243490593155d09870b7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41367 } }
I20260620 08:08:29.775359 20482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69fba2a84243490593155d09870b7162. Latest consensus state: current_term: 1 leader_uuid: "69fba2a84243490593155d09870b7162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69fba2a84243490593155d09870b7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41367 } } }
I20260620 08:08:29.775600 20483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69fba2a84243490593155d09870b7162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69fba2a84243490593155d09870b7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41367 } } }
I20260620 08:08:29.775657 20482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:29.775663 20483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162 [sys.catalog]: This master's current role is: LEADER
W20260620 08:08:29.775990 20495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:29.776090 20495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:29.776237 20497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:29.776482 20497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:29.777714 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:29.770235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38675 (local address 127.0.0.1:41367)
0620 08:08:29.770491 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:29.770502 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:29.770546 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:08:29.771122 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:29.771129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:29.771186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:29.771389 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:29.771588 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:29.773469 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:29.774945 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:29.775800 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:29.775964 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:29.776760 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:29.776764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:29.776764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:29.776985 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:29.777023 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:29.777039 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:29.777123 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:29.777346 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 08:08:29.777438 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260620 08:08:29.778466 20497 catalog_manager.cc:1347] Generated new cluster ID: 92400babf48f47ac804ddedcef70c9e2
I20260620 08:08:29.778563 20497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:29.778780 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:29.784950 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:29.780285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43235 (local address 127.0.0.1:41367)
0620 08:08:29.780330 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:29.780332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:29.781965 (+ 1633us) server_negotiation.cc:407] Connection header received
0620 08:08:29.782052 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:29.782054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:29.782068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:29.782132 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:29.782302 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:29.782787 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:29.783553 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:29.783768 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:29.783861 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:29.784633 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:29.784634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:29.784635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:29.784640 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:29.784658 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:29.784663 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:29.784739 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:29.784808 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 08:08:29.784837 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:08:29.805335 20497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:29.806160 20497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:29.814889 20497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69fba2a84243490593155d09870b7162: Generated new TSK 0
I20260620 08:08:29.815121 20497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:29.821563 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20408
2026-06-20T08:08:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:29 dist-test-slave-c4w8 krb5kdc[20507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:29 dist-test-slave-c4w8 krb5kdc[20507](info): set up 2 sockets
Jun 20 08:08:29 dist-test-slave-c4w8 krb5kdc[20507](info): commencing operation
krb5kdc: starting...
W20260620 08:08:30.878520 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:30 dist-test-slave-c4w8 krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:31.021512 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:31.107331 20520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:31.107605 20520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:31.107631 20520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:31.108911 20520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:31.108968 20520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:31.108983 20520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:31.108996 20520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:31.109009 20520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:31.110661 20520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:31.110957 20520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:31.111212 20520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:31.116979 20528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:31.116992 20526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:31.116972 20525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:31.117357 20520 server_base.cc:1034] running on GCE node
I20260620 08:08:31.117666 20520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:31.117982 20520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:31.119186 20520 hybrid_clock.cc:648] HybridClock initialized: now 1781942911119163 us; error 46 us; skew 500 ppm
Jun 20 08:08:31 dist-test-slave-c4w8 krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:31.127996 20520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:31.128541 20520 webserver.cc:460] Webserver started at http://127.0.0.1:36125/ using document root <none> and password file <none>
I20260620 08:08:31.128785 20520 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:31.128854 20520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:31.129020 20520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:31.130254 20520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "fb43f44b47544ad093e257e2522d736f"
format_stamp: "Formatted at 2026-06-20 08:08:31 on dist-test-slave-c4w8"
server_key: "9ec2c74daeb36355a1796a805a3d28e9"
server_key_iv: "2e637c482d45938f39844a7ac3bc3763"
server_key_version: "encryptionkey@0"
I20260620 08:08:31.130703 20520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "fb43f44b47544ad093e257e2522d736f"
format_stamp: "Formatted at 2026-06-20 08:08:31 on dist-test-slave-c4w8"
server_key: "9ec2c74daeb36355a1796a805a3d28e9"
server_key_iv: "2e637c482d45938f39844a7ac3bc3763"
server_key_version: "encryptionkey@0"
I20260620 08:08:31.132241 20520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:31.133158 20535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:31.133451 20520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:31.133582 20520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "fb43f44b47544ad093e257e2522d736f"
format_stamp: "Formatted at 2026-06-20 08:08:31 on dist-test-slave-c4w8"
server_key: "9ec2c74daeb36355a1796a805a3d28e9"
server_key_iv: "2e637c482d45938f39844a7ac3bc3763"
server_key_version: "encryptionkey@0"
I20260620 08:08:31.133718 20520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:31.137279 20520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:31.139739 20520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:31.139940 20520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:31.144390 20520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46797
I20260620 08:08:31.144425 20587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46797 every 8 connection(s)
I20260620 08:08:31.144783 20520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:31.145944 20588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:31.147464 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20520
I20260620 08:08:31.147599 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:31.147807 32207 external_mini_cluster.cc:1428] Setting key b4e8ed678499497f8b5340aa701702c3
I20260620 08:08:31.149175 20588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f: Bootstrap starting.
I20260620 08:08:31.149974 20588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:31.150342 20588 log.cc:824] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:31.150723 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:31.148407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33655 (local address 127.0.0.1:46797)
0620 08:08:31.148631 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:31.148642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:31.148679 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:08:31.149266 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:31.149275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:31.149311 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:31.149464 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:31.150236 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:31.150241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:31.150242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:31.150247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:31.150295 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:31.150310 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:31.150404 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:31.150491 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 08:08:31.150538 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260620 08:08:31.151263 20588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f: No bootstrap required, opened a new log
I20260620 08:08:31.151841 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:31.154139 20588 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb43f44b47544ad093e257e2522d736f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260620 08:08:31.154357 20588 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:31.154400 20588 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb43f44b47544ad093e257e2522d736f, State: Initialized, Role: FOLLOWER
I20260620 08:08:31.154531 20588 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb43f44b47544ad093e257e2522d736f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260620 08:08:31.154644 20588 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:31.154731 20588 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:31.154799 20588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:31.154889 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:31.153509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42421 (local address 127.0.0.1:46797)
0620 08:08:31.153554 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:31.153557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:31.153716 (+ 159us) server_negotiation.cc:407] Connection header received
0620 08:08:31.153823 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:31.153824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:31.153829 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:31.153871 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:31.154583 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:31.154584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:31.154585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:31.154588 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:31.154602 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:31.154605 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:31.154681 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:31.154738 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 08:08:31.154752 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:31.155586 20588 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb43f44b47544ad093e257e2522d736f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260620 08:08:31.155681 20588 leader_election.cc:302] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb43f44b47544ad093e257e2522d736f; no voters:
I20260620 08:08:31.155855 20588 leader_election.cc:288] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:31.155974 20598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:31.156111 20588 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:31.156152 20598 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [term 1 LEADER]: Becoming Leader. State: Replica: fb43f44b47544ad093e257e2522d736f, State: Running, Role: LEADER
I20260620 08:08:31.156237 20598 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb43f44b47544ad093e257e2522d736f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260620 08:08:31.156653 20599 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb43f44b47544ad093e257e2522d736f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb43f44b47544ad093e257e2522d736f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } } }
I20260620 08:08:31.156653 20600 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb43f44b47544ad093e257e2522d736f. Latest consensus state: current_term: 1 leader_uuid: "fb43f44b47544ad093e257e2522d736f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb43f44b47544ad093e257e2522d736f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } } }
I20260620 08:08:31.156777 20600 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:31.156773 20599 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:31.157028 20607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:31.157564 20607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:31.159224 20607 catalog_manager.cc:1347] Generated new cluster ID: 5b60afd2a45c4c51abff6409da97d1b5
I20260620 08:08:31.159277 20607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:31.163247 20607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:31.163477 20607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:31.166436 20607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb43f44b47544ad093e257e2522d736f: Generated new TSK 0
I20260620 08:08:31.166635 20607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:31.176932 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20520
2026-06-20T08:08:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:31 dist-test-slave-c4w8 krb5kdc[20619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:31 dist-test-slave-c4w8 krb5kdc[20619](info): set up 2 sockets
Jun 20 08:08:31 dist-test-slave-c4w8 krb5kdc[20619](info): commencing operation
krb5kdc: starting...
W20260620 08:08:32.238289 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:32 dist-test-slave-c4w8 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:32.368974 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:32.456880 20632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:32.457132 20632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:32.457160 20632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:32.458655 20632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:32.458722 20632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:32.458740 20632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:32.458756 20632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:32.458770 20632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:32.460572 20632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:32.460886 20632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:32.461303 20632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:32.467003 20638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:32.467017 20640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:32.467064 20637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:32.467129 20632 server_base.cc:1034] running on GCE node
I20260620 08:08:32.467579 20632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:32.467964 20632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:32.469151 20632 hybrid_clock.cc:648] HybridClock initialized: now 1781942912469118 us; error 52 us; skew 500 ppm
Jun 20 08:08:32 dist-test-slave-c4w8 krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:32.472009 20632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:32.472541 20632 webserver.cc:460] Webserver started at http://127.0.0.1:45043/ using document root <none> and password file <none>
I20260620 08:08:32.472819 20632 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:32.472891 20632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:32.473052 20632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:32.474299 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "49a2df2b2c0e4551a4b82dab8c923e44"
format_stamp: "Formatted at 2026-06-20 08:08:32 on dist-test-slave-c4w8"
server_key: "b233729d18416d60ffcd2db07166bf12"
server_key_iv: "dfed494ec3e1b0edd58318959edb15ad"
server_key_version: "encryptionkey@0"
I20260620 08:08:32.474681 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "49a2df2b2c0e4551a4b82dab8c923e44"
format_stamp: "Formatted at 2026-06-20 08:08:32 on dist-test-slave-c4w8"
server_key: "b233729d18416d60ffcd2db07166bf12"
server_key_iv: "dfed494ec3e1b0edd58318959edb15ad"
server_key_version: "encryptionkey@0"
I20260620 08:08:32.476136 20632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:32.477214 20647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:32.477582 20632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:32.477768 20632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "49a2df2b2c0e4551a4b82dab8c923e44"
format_stamp: "Formatted at 2026-06-20 08:08:32 on dist-test-slave-c4w8"
server_key: "b233729d18416d60ffcd2db07166bf12"
server_key_iv: "dfed494ec3e1b0edd58318959edb15ad"
server_key_version: "encryptionkey@0"
I20260620 08:08:32.477856 20632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:32.501061 20632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:32.503763 20632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:32.503962 20632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:32.508451 20632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44353
I20260620 08:08:32.508504 20699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44353 every 8 connection(s)
I20260620 08:08:32.508922 20632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:32.510105 20700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:32.512817 20700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44: Bootstrap starting.
I20260620 08:08:32.513523 20700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:32.513904 20700 log.cc:824] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:32.514782 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20632
I20260620 08:08:32.514912 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:32.514945 20700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44: No bootstrap required, opened a new log
I20260620 08:08:32.515136 32207 external_mini_cluster.cc:1428] Setting key 981958b7326b474ad5e7079a5b4c9538
I20260620 08:08:32.517424 20700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 08:08:32.517632 20700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:32.517678 20700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a2df2b2c0e4551a4b82dab8c923e44, State: Initialized, Role: FOLLOWER
I20260620 08:08:32.517819 20700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 08:08:32.517925 20700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:32.517966 20700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:32.518011 20700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:32.518749 20700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 08:08:32.518869 20700 leader_election.cc:302] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49a2df2b2c0e4551a4b82dab8c923e44; no voters:
I20260620 08:08:32.519009 20700 leader_election.cc:288] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:32.519079 20705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:32.519258 20700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:32.519260 20705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [term 1 LEADER]: Becoming Leader. State: Replica: 49a2df2b2c0e4551a4b82dab8c923e44, State: Running, Role: LEADER
I20260620 08:08:32.519485 20705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260620 08:08:32.519882 20705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a2df2b2c0e4551a4b82dab8c923e44. Latest consensus state: current_term: 1 leader_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260620 08:08:32.520092 20705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:32.520022 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a2df2b2c0e4551a4b82dab8c923e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260620 08:08:32.520526 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:32.520716 20714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:32.521328 20714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:32.523162 20714 catalog_manager.cc:1347] Generated new cluster ID: b84b9f8f4a2843d0b27e97c6ea9056d7
I20260620 08:08:32.523221 20714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:08:32 dist-test-slave-c4w8 krb5kdc[20619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:08:32.527554 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:32.515828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:48477 (local address 127.0.0.1:44353)
0620 08:08:32.516046 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:32.516057 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:32.516097 (+ 40us) server_negotiation.cc:407] Connection header received
0620 08:08:32.516581 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:32.516592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:32.516657 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:32.516811 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:32.517465 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:32.519200 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:32.520398 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:32.521044 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:32.521173 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:32.524059 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:32.524063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:32.524064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:32.524076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:32.525951 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:32.526367 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:32.526370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:32.526371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:32.526463 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:32.526698 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:32.526700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:32.526700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:32.526954 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:32.527157 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:32.527244 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 08:08:32.527348 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260620 08:08:32.528507 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:32.536340 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:32.529910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:39897 (local address 127.0.0.1:44353)
0620 08:08:32.529949 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:32.529951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:32.530117 (+ 166us) server_negotiation.cc:407] Connection header received
0620 08:08:32.530176 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:32.530177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:32.530188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:32.530243 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:32.530766 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:32.531278 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:32.532055 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:32.532306 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:32.532399 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:32.533679 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:32.533680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:32.533681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:32.533688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:32.535120 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:32.535510 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:32.535511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:32.535512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:32.535571 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:32.535792 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:32.535794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:32.535794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:32.535912 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:32.536043 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:32.536154 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 08:08:32.536219 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:08:32.553278 20714 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:32.553946 20714 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:32.558780 20714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49a2df2b2c0e4551a4b82dab8c923e44: Generated new TSK 0
I20260620 08:08:32.559029 20714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:32.561656 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20632
W20260620 08:08:32.566546 20617 connection.cc:535] client connection to 127.0.0.1:44353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44353): Connection reset by peer (error 104)
2026-06-20T08:08:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:32 dist-test-slave-c4w8 krb5kdc[20731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:32 dist-test-slave-c4w8 krb5kdc[20731](info): set up 2 sockets
Jun 20 08:08:32 dist-test-slave-c4w8 krb5kdc[20731](info): commencing operation
krb5kdc: starting...
W20260620 08:08:33.618399 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:33 dist-test-slave-c4w8 krb5kdc[20731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:33.746955 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:33.831270 20744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:33.831501 20744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:33.831527 20744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:33.832928 20744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:33.833019 20744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:33.833043 20744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:33.833066 20744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:33.833106 20744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:33.835245 20744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:33.835558 20744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:33.835837 20744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:33.841325 20749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:33.841367 20744 server_base.cc:1034] running on GCE node
W20260620 08:08:33.841367 20750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:33.841320 20752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:33.841869 20744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:33.842223 20744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:33.843423 20744 hybrid_clock.cc:648] HybridClock initialized: now 1781942913843401 us; error 45 us; skew 500 ppm
Jun 20 08:08:33 dist-test-slave-c4w8 krb5kdc[20731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:33.846369 20744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:33.846957 20744 webserver.cc:460] Webserver started at http://127.0.0.1:39805/ using document root <none> and password file <none>
I20260620 08:08:33.847193 20744 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:33.847261 20744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:33.847437 20744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:33.848531 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "377ffdfe8cd64f9b920534c4a8bfb364"
format_stamp: "Formatted at 2026-06-20 08:08:33 on dist-test-slave-c4w8"
server_key: "d83564714a8689caa4015857a1304dbe"
server_key_iv: "64b443d28167636ea0ab9d007a49ed0d"
server_key_version: "encryptionkey@0"
I20260620 08:08:33.848902 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "377ffdfe8cd64f9b920534c4a8bfb364"
format_stamp: "Formatted at 2026-06-20 08:08:33 on dist-test-slave-c4w8"
server_key: "d83564714a8689caa4015857a1304dbe"
server_key_iv: "64b443d28167636ea0ab9d007a49ed0d"
server_key_version: "encryptionkey@0"
I20260620 08:08:33.850493 20744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:33.851495 20759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:33.851729 20744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:33.851847 20744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "377ffdfe8cd64f9b920534c4a8bfb364"
format_stamp: "Formatted at 2026-06-20 08:08:33 on dist-test-slave-c4w8"
server_key: "d83564714a8689caa4015857a1304dbe"
server_key_iv: "64b443d28167636ea0ab9d007a49ed0d"
server_key_version: "encryptionkey@0"
I20260620 08:08:33.851939 20744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:33.865453 20744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:33.867988 20744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:33.868247 20744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:33.872885 20744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44305
I20260620 08:08:33.872918 20811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44305 every 8 connection(s)
I20260620 08:08:33.873328 20744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:33.874315 20812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:33.876657 20812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364: Bootstrap starting.
I20260620 08:08:33.877218 20812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:33.877465 20812 log.cc:824] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:33.878247 20812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364: No bootstrap required, opened a new log
I20260620 08:08:33.879892 20812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44305 } }
I20260620 08:08:33.880043 20812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:33.880065 20812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377ffdfe8cd64f9b920534c4a8bfb364, State: Initialized, Role: FOLLOWER
I20260620 08:08:33.880172 20812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44305 } }
I20260620 08:08:33.880225 20812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:33.880246 20812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:33.880334 20812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:33.880949 20812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44305 } }
I20260620 08:08:33.881042 20812 leader_election.cc:302] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 377ffdfe8cd64f9b920534c4a8bfb364; no voters:
I20260620 08:08:33.881245 20812 leader_election.cc:288] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:33.881369 20815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:33.881543 20812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:33.881647 20815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [term 1 LEADER]: Becoming Leader. State: Replica: 377ffdfe8cd64f9b920534c4a8bfb364, State: Running, Role: LEADER
I20260620 08:08:33.881765 20815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44305 } }
I20260620 08:08:33.882447 20817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 377ffdfe8cd64f9b920534c4a8bfb364. Latest consensus state: current_term: 1 leader_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44305 } } }
I20260620 08:08:33.882572 20817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:33.882642 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ffdfe8cd64f9b920534c4a8bfb364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44305 } } }
I20260620 08:08:33.882714 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:33.882907 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20744
I20260620 08:08:33.883050 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:33.883297 32207 external_mini_cluster.cc:1428] Setting key f21f4e5b60aca3e08e2b727d8b1a6794
W20260620 08:08:33.883621 20830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:33.883734 20830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:33.883916 20827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:33.884099 20827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:33.885761 20827 catalog_manager.cc:1347] Generated new cluster ID: ea74c02036e9409c8f0dda6c0577096d
I20260620 08:08:33.885813 20827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:33.890810 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:33.883825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:49623 (local address 127.0.0.1:44305)
0620 08:08:33.884771 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:33.884781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:33.884812 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:08:33.884927 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:33.884934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:33.884992 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:33.885136 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:33.885501 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:33.887000 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:33.888264 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:33.888934 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:33.889058 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:33.889992 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:33.889997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:33.889998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:33.890260 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:33.890315 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:33.890349 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:33.890442 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:33.890545 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:08:33.890578 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":75,"threads_started":1}
I20260620 08:08:33.891923 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:33.897891 20827 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:33.898702 20827 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:33.901815 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:33.894024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:34167 (local address 127.0.0.1:44305)
0620 08:08:33.894065 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:33.894067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:33.897401 (+ 3334us) server_negotiation.cc:407] Connection header received
0620 08:08:33.897720 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:33.897721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:33.897748 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:33.897808 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:33.898063 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:33.898851 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:33.900153 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:33.900496 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:33.900579 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:33.901406 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:33.901407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:33.901407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:33.901413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:33.901429 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:33.901434 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:33.901497 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:33.901712 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 08:08:33.901729 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:33.904606 20827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 377ffdfe8cd64f9b920534c4a8bfb364: Generated new TSK 0
I20260620 08:08:33.904814 20827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:33.907922 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20744
W20260620 08:08:33.912672 20729 connection.cc:535] client connection to 127.0.0.1:44305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44305): Connection reset by peer (error 104)
2026-06-20T08:08:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:33 dist-test-slave-c4w8 krb5kdc[20843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:33 dist-test-slave-c4w8 krb5kdc[20843](info): set up 2 sockets
Jun 20 08:08:33 dist-test-slave-c4w8 krb5kdc[20843](info): commencing operation
krb5kdc: starting...
W20260620 08:08:34.962311 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:35 dist-test-slave-c4w8 krb5kdc[20843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:35.093890 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:35.181538 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:35.181877 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:35.181919 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:35.183243 20856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:35.183322 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:35.183350 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:35.183383 20856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:35.183411 20856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:35.184973 20856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:35.185261 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:35.185498 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:35.190441 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:35.190490 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:35.190446 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:35.190492 20856 server_base.cc:1034] running on GCE node
I20260620 08:08:35.190969 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:35.191279 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:35.192489 20856 hybrid_clock.cc:648] HybridClock initialized: now 1781942915192459 us; error 45 us; skew 500 ppm
Jun 20 08:08:35 dist-test-slave-c4w8 krb5kdc[20843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:35.195343 20856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:35.195828 20856 webserver.cc:460] Webserver started at http://127.0.0.1:39105/ using document root <none> and password file <none>
I20260620 08:08:35.196080 20856 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:35.196156 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:35.196326 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:35.197510 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "38fc436503504211807b9aeb233dd648"
format_stamp: "Formatted at 2026-06-20 08:08:35 on dist-test-slave-c4w8"
server_key: "c58f4d4c6629a9eab106aeb2b0f3d6b8"
server_key_iv: "0916fc7b5380f10358622ee236a898ca"
server_key_version: "encryptionkey@0"
I20260620 08:08:35.197990 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "38fc436503504211807b9aeb233dd648"
format_stamp: "Formatted at 2026-06-20 08:08:35 on dist-test-slave-c4w8"
server_key: "c58f4d4c6629a9eab106aeb2b0f3d6b8"
server_key_iv: "0916fc7b5380f10358622ee236a898ca"
server_key_version: "encryptionkey@0"
I20260620 08:08:35.199678 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:35.200628 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:35.200940 20856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:35.201092 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "38fc436503504211807b9aeb233dd648"
format_stamp: "Formatted at 2026-06-20 08:08:35 on dist-test-slave-c4w8"
server_key: "c58f4d4c6629a9eab106aeb2b0f3d6b8"
server_key_iv: "0916fc7b5380f10358622ee236a898ca"
server_key_version: "encryptionkey@0"
I20260620 08:08:35.201195 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:35.205080 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:35.208324 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:35.208534 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:35.213740 20856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40857
I20260620 08:08:35.213773 20923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40857 every 8 connection(s)
I20260620 08:08:35.214207 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:35.215224 20924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:35.217902 20924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648: Bootstrap starting.
I20260620 08:08:35.218545 20924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:35.218850 20924 log.cc:824] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:35.219523 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20856
I20260620 08:08:35.219655 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:35.219802 20924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648: No bootstrap required, opened a new log
I20260620 08:08:35.219856 32207 external_mini_cluster.cc:1428] Setting key efa567664c0383c09b2c84989ad9fc92
I20260620 08:08:35.222554 20924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc436503504211807b9aeb233dd648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 08:08:35.222743 20924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:35.222810 20924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fc436503504211807b9aeb233dd648, State: Initialized, Role: FOLLOWER
I20260620 08:08:35.222994 20924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc436503504211807b9aeb233dd648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 08:08:35.223079 20924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:35.223104 20924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:35.223178 20924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:35.223341 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:35.220527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:36971 (local address 127.0.0.1:40857)
0620 08:08:35.220746 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:35.220755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:35.220784 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:08:35.221469 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:35.221477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:35.221508 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:35.221719 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:35.222555 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:35.222559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:35.222560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:35.222565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:35.222607 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:35.222619 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:35.222709 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:35.222784 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:08:35.223019 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 08:08:35.224993 20924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc436503504211807b9aeb233dd648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 08:08:35.225093 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:35.225145 20924 leader_election.cc:302] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fc436503504211807b9aeb233dd648; no voters:
I20260620 08:08:35.225322 20924 leader_election.cc:288] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:35.225457 20929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:35.225715 20924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:35.225921 20929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [term 1 LEADER]: Becoming Leader. State: Replica: 38fc436503504211807b9aeb233dd648, State: Running, Role: LEADER
I20260620 08:08:35.226069 20929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc436503504211807b9aeb233dd648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260620 08:08:35.226491 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fc436503504211807b9aeb233dd648. Latest consensus state: current_term: 1 leader_uuid: "38fc436503504211807b9aeb233dd648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc436503504211807b9aeb233dd648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } } }
I20260620 08:08:35.226501 20930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fc436503504211807b9aeb233dd648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fc436503504211807b9aeb233dd648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } } }
I20260620 08:08:35.226670 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:35.226678 20930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:35.227010 20943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:35.227663 20943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:35.228569 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:35.226806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:58037 (local address 127.0.0.1:40857)
0620 08:08:35.226846 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:35.226848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:35.227505 (+ 657us) server_negotiation.cc:407] Connection header received
0620 08:08:35.227614 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:35.227615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:35.227622 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:35.227680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:35.228343 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:35.228344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:35.228345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:35.228350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:35.228367 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:35.228373 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:35.228426 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:35.228493 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:08:35.228507 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:08:35.229812 20943 catalog_manager.cc:1347] Generated new cluster ID: a31732d76f3f4bf4b39f465db80c95ee
I20260620 08:08:35.229882 20943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:35.258293 20943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:35.258510 20943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:35.263530 20943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38fc436503504211807b9aeb233dd648: Generated new TSK 0
I20260620 08:08:35.263767 20943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:35.269354 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20856
2026-06-20T08:08:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:35 dist-test-slave-c4w8 krb5kdc[20955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:35 dist-test-slave-c4w8 krb5kdc[20955](info): set up 2 sockets
Jun 20 08:08:35 dist-test-slave-c4w8 krb5kdc[20955](info): commencing operation
krb5kdc: starting...
W20260620 08:08:36.330868 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:36 dist-test-slave-c4w8 krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:36.458088 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:36.542232 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:36.542495 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:36.542536 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:36.543783 20968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:36.543864 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:36.543888 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:36.543923 20968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:36.543951 20968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:36.545513 20968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:36.545842 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:36.546094 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:36.551256 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:36.551277 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:36.551270 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:36.551677 20968 server_base.cc:1034] running on GCE node
I20260620 08:08:36.551900 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:36.552165 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:36.553362 20968 hybrid_clock.cc:648] HybridClock initialized: now 1781942916553344 us; error 33 us; skew 500 ppm
Jun 20 08:08:36 dist-test-slave-c4w8 krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:36.560459 20968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:36.560966 20968 webserver.cc:460] Webserver started at http://127.0.0.1:42409/ using document root <none> and password file <none>
I20260620 08:08:36.561187 20968 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:36.561228 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:36.561328 20968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:36.562446 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "5cce1bff7ddd4e2784fd947fcafd0be7"
format_stamp: "Formatted at 2026-06-20 08:08:36 on dist-test-slave-c4w8"
server_key: "c062e4aff00d66c3b509caba48c2854b"
server_key_iv: "f5f04ff58105c59469105d378ffc9f09"
server_key_version: "encryptionkey@0"
I20260620 08:08:36.562745 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "5cce1bff7ddd4e2784fd947fcafd0be7"
format_stamp: "Formatted at 2026-06-20 08:08:36 on dist-test-slave-c4w8"
server_key: "c062e4aff00d66c3b509caba48c2854b"
server_key_iv: "f5f04ff58105c59469105d378ffc9f09"
server_key_version: "encryptionkey@0"
I20260620 08:08:36.564106 20968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:08:36.564908 20983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:36.565156 20968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:36.565235 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "5cce1bff7ddd4e2784fd947fcafd0be7"
format_stamp: "Formatted at 2026-06-20 08:08:36 on dist-test-slave-c4w8"
server_key: "c062e4aff00d66c3b509caba48c2854b"
server_key_iv: "f5f04ff58105c59469105d378ffc9f09"
server_key_version: "encryptionkey@0"
I20260620 08:08:36.565351 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:36.580413 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:36.582834 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:36.583002 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:36.587860 20968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39903
I20260620 08:08:36.587888 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39903 every 8 connection(s)
I20260620 08:08:36.588271 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:36.589190 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:36.591605 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7: Bootstrap starting.
I20260620 08:08:36.592178 21036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:36.592430 21036 log.cc:824] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:36.593149 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7: No bootstrap required, opened a new log
I20260620 08:08:36.593590 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 20968
I20260620 08:08:36.593787 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:36.594010 32207 external_mini_cluster.cc:1428] Setting key ea48ce85da274ce99f23e09062e8af61
I20260620 08:08:36.595481 21036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 08:08:36.595724 21036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:36.595786 21036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cce1bff7ddd4e2784fd947fcafd0be7, State: Initialized, Role: FOLLOWER
I20260620 08:08:36.595969 21036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 08:08:36.596074 21036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:36.596123 21036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:36.596170 21036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:36.596817 21036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 08:08:36.596948 21036 leader_election.cc:302] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cce1bff7ddd4e2784fd947fcafd0be7; no voters:
I20260620 08:08:36.597170 21036 leader_election.cc:288] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:36.597220 21041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:36.597355 21041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [term 1 LEADER]: Becoming Leader. State: Replica: 5cce1bff7ddd4e2784fd947fcafd0be7, State: Running, Role: LEADER
I20260620 08:08:36.597500 21036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:36.597517 21041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 08:08:36.598016 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } } }
I20260620 08:08:36.598121 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:36.598315 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cce1bff7ddd4e2784fd947fcafd0be7. Latest consensus state: current_term: 1 leader_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cce1bff7ddd4e2784fd947fcafd0be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } } }
I20260620 08:08:36.598393 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7 [sys.catalog]: This master's current role is: LEADER
W20260620 08:08:36.599633 21057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:36.599699 21057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:36.599749 21049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:36.599920 21049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:36.601871 21049 catalog_manager.cc:1347] Generated new cluster ID: 3792538b80a7418a95180e60f9b6a30c
I20260620 08:08:36.601933 21049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:08:36 dist-test-slave-c4w8 krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:08:36.607514 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:36.594639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:50261 (local address 127.0.0.1:39903)
0620 08:08:36.594826 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:36.594835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:36.594863 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:08:36.595622 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:36.595629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:36.595685 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:36.595868 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:36.596670 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:36.598061 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:36.599300 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:36.600007 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:36.600120 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:36.603433 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:36.603438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:36.603439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:36.603453 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:36.605631 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:36.606118 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:36.606123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:36.606123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:36.606255 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:36.606522 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:36.606523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:36.606523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:36.606869 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:36.607068 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:36.607178 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:08:36.607293 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260620 08:08:36.608734 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:36.618409 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:36.610289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:58489 (local address 127.0.0.1:39903)
0620 08:08:36.610329 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:36.610330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:36.610618 (+ 288us) server_negotiation.cc:407] Connection header received
0620 08:08:36.610653 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:36.610654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:36.610666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:36.610750 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:36.611418 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:36.612239 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:36.613301 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:36.613708 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:36.613795 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:36.615291 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:36.615293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:36.615294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:36.615302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:36.617022 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:36.617479 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:36.617480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:36.617480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:36.617545 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:36.617814 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:36.617815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:36.617815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:36.617965 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:36.618085 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:36.618232 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:08:36.618303 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:08:36.621304 21049 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:36.622046 21049 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:36.628144 21049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cce1bff7ddd4e2784fd947fcafd0be7: Generated new TSK 0
I20260620 08:08:36.628333 21049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:36.633693 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 20968
W20260620 08:08:36.638389 20953 connection.cc:535] client connection to 127.0.0.1:39903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39903): Connection reset by peer (error 104)
2026-06-20T08:08:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:36 dist-test-slave-c4w8 krb5kdc[21067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:36 dist-test-slave-c4w8 krb5kdc[21067](info): set up 2 sockets
Jun 20 08:08:36 dist-test-slave-c4w8 krb5kdc[21067](info): commencing operation
krb5kdc: starting...
W20260620 08:08:37.690968 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:37 dist-test-slave-c4w8 krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:37.818153 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:37.905406 21080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:37.905697 21080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:37.905745 21080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:37.906955 21080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:37.907037 21080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:37.907068 21080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:37.907105 21080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:37.907138 21080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:37.908727 21080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:37.909036 21080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:37.909312 21080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:37.914939 21088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:37.914901 21086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:37.914988 21080 server_base.cc:1034] running on GCE node
W20260620 08:08:37.914884 21085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:37.915457 21080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:37.915737 21080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:37.916936 21080 hybrid_clock.cc:648] HybridClock initialized: now 1781942917916897 us; error 60 us; skew 500 ppm
Jun 20 08:08:37 dist-test-slave-c4w8 krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:37.919883 21080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:37.920379 21080 webserver.cc:460] Webserver started at http://127.0.0.1:45567/ using document root <none> and password file <none>
I20260620 08:08:37.920624 21080 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:37.920691 21080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:37.920852 21080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:37.922043 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "210e90ccec41432a916048c2d4a65f50"
format_stamp: "Formatted at 2026-06-20 08:08:37 on dist-test-slave-c4w8"
server_key: "dacb11c4c7e83690c924c81a7f79a949"
server_key_iv: "0b7cc410b04c1c2814e10a1a6afc075c"
server_key_version: "encryptionkey@0"
I20260620 08:08:37.922423 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "210e90ccec41432a916048c2d4a65f50"
format_stamp: "Formatted at 2026-06-20 08:08:37 on dist-test-slave-c4w8"
server_key: "dacb11c4c7e83690c924c81a7f79a949"
server_key_iv: "0b7cc410b04c1c2814e10a1a6afc075c"
server_key_version: "encryptionkey@0"
I20260620 08:08:37.923890 21080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:37.924703 21095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:37.924947 21080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:37.925066 21080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "210e90ccec41432a916048c2d4a65f50"
format_stamp: "Formatted at 2026-06-20 08:08:37 on dist-test-slave-c4w8"
server_key: "dacb11c4c7e83690c924c81a7f79a949"
server_key_iv: "0b7cc410b04c1c2814e10a1a6afc075c"
server_key_version: "encryptionkey@0"
I20260620 08:08:37.925143 21080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:37.952535 21080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:37.955040 21080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:37.955220 21080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:37.959728 21080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33949
I20260620 08:08:37.959759 21147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33949 every 8 connection(s)
I20260620 08:08:37.960229 21080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:37.961234 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:37.963619 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50: Bootstrap starting.
I20260620 08:08:37.963958 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21080
I20260620 08:08:37.964118 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:37.964315 21148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:37.964342 32207 external_mini_cluster.cc:1428] Setting key f0e13beeedc21cbae30ee23055538363
I20260620 08:08:37.964617 21148 log.cc:824] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:37.965586 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50: No bootstrap required, opened a new log
I20260620 08:08:37.968385 21148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e90ccec41432a916048c2d4a65f50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33949 } }
I20260620 08:08:37.968541 21148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:37.968566 21148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210e90ccec41432a916048c2d4a65f50, State: Initialized, Role: FOLLOWER
I20260620 08:08:37.968716 21148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e90ccec41432a916048c2d4a65f50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33949 } }
I20260620 08:08:37.968802 21148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:37.968858 21148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:37.968912 21148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:37.969803 21148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e90ccec41432a916048c2d4a65f50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33949 } }
I20260620 08:08:37.969930 21148 leader_election.cc:302] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210e90ccec41432a916048c2d4a65f50; no voters:
I20260620 08:08:37.970173 21148 leader_election.cc:288] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:37.970240 21153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:37.970432 21153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [term 1 LEADER]: Becoming Leader. State: Replica: 210e90ccec41432a916048c2d4a65f50, State: Running, Role: LEADER
I20260620 08:08:37.970474 21148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:37.970541 21153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e90ccec41432a916048c2d4a65f50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33949 } }
I20260620 08:08:37.970925 21154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210e90ccec41432a916048c2d4a65f50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e90ccec41432a916048c2d4a65f50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33949 } } }
I20260620 08:08:37.971016 21154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:37.971182 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210e90ccec41432a916048c2d4a65f50. Latest consensus state: current_term: 1 leader_uuid: "210e90ccec41432a916048c2d4a65f50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210e90ccec41432a916048c2d4a65f50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33949 } } }
I20260620 08:08:37.971362 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:37.971807 21159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:37.972131 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:37.964926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:53637 (local address 127.0.0.1:33949)
0620 08:08:37.965145 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:37.965156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:37.965193 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:08:37.965914 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:37.965923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:37.965994 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:37.966168 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:37.966404 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:37.968035 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:37.969117 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:37.969875 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:37.970003 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:37.970999 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:37.971004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:37.971005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:37.971315 (+ 310us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:37.971356 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:37.971375 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:37.971760 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:37.971820 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 08:08:37.971860 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260620 08:08:37.972373 21159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:37.973122 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:37.974241 21159 catalog_manager.cc:1347] Generated new cluster ID: faa1f4a6b9aa48689e459450ca9cbc23
I20260620 08:08:37.974298 21159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:37.981709 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:37.978182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:51077 (local address 127.0.0.1:33949)
0620 08:08:37.978227 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:37.978229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:37.978461 (+ 232us) server_negotiation.cc:407] Connection header received
0620 08:08:37.978534 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:37.978534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:37.978546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:37.978621 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:37.978798 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:37.979603 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:37.980346 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:37.980575 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:37.980655 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:37.981401 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:37.981402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:37.981403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:37.981408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:37.981430 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:37.981435 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:37.981499 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:37.981566 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:08:37.981576 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:37.986922 21159 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:37.987521 21159 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:37.991564 21159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210e90ccec41432a916048c2d4a65f50: Generated new TSK 0
I20260620 08:08:37.991770 21159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:37.994752 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21080
W20260620 08:08:37.999358 21065 connection.cc:535] client connection to 127.0.0.1:33949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33949): Connection reset by peer (error 104)
2026-06-20T08:08:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:38 dist-test-slave-c4w8 krb5kdc[21179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:38 dist-test-slave-c4w8 krb5kdc[21179](info): set up 2 sockets
Jun 20 08:08:38 dist-test-slave-c4w8 krb5kdc[21179](info): commencing operation
krb5kdc: starting...
W20260620 08:08:39.050344 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:39 dist-test-slave-c4w8 krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:08:39.182631 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:39.265447 21192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:39.265719 21192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:39.265746 21192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:39.266978 21192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:39.267022 21192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:39.267038 21192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:39.267051 21192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:39.267063 21192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:39.268620 21192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:39.268850 21192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:39.269042 21192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:39.274715 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:39.274719 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:39.274727 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:39.274967 21192 server_base.cc:1034] running on GCE node
I20260620 08:08:39.275252 21192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:39.275574 21192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:39.276746 21192 hybrid_clock.cc:648] HybridClock initialized: now 1781942919276706 us; error 60 us; skew 500 ppm
Jun 20 08:08:39 dist-test-slave-c4w8 krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:39.279613 21192 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:08:39.280174 21192 webserver.cc:460] Webserver started at http://127.0.0.1:33245/ using document root <none> and password file <none>
I20260620 08:08:39.280445 21192 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:39.280521 21192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:39.280673 21192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:39.281849 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "d05b27ebe09b4b1089d4653db6c5ffcd"
format_stamp: "Formatted at 2026-06-20 08:08:39 on dist-test-slave-c4w8"
server_key: "f58afdebdb8a16635aa237ad4bb678c9"
server_key_iv: "65aaa1d40071be59d8fc03266e598dab"
server_key_version: "encryptionkey@0"
I20260620 08:08:39.282215 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "d05b27ebe09b4b1089d4653db6c5ffcd"
format_stamp: "Formatted at 2026-06-20 08:08:39 on dist-test-slave-c4w8"
server_key: "f58afdebdb8a16635aa237ad4bb678c9"
server_key_iv: "65aaa1d40071be59d8fc03266e598dab"
server_key_version: "encryptionkey@0"
I20260620 08:08:39.283666 21192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:39.284519 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:39.284798 21192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:39.284919 21192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "d05b27ebe09b4b1089d4653db6c5ffcd"
format_stamp: "Formatted at 2026-06-20 08:08:39 on dist-test-slave-c4w8"
server_key: "f58afdebdb8a16635aa237ad4bb678c9"
server_key_iv: "65aaa1d40071be59d8fc03266e598dab"
server_key_version: "encryptionkey@0"
I20260620 08:08:39.284998 21192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:39.288635 21192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:39.290962 21192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:39.291126 21192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:39.295420 21192 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32945
I20260620 08:08:39.295534 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32945 every 8 connection(s)
I20260620 08:08:39.295850 21192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:39.296916 21260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:39.298060 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21192
I20260620 08:08:39.298194 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:39.298380 32207 external_mini_cluster.cc:1428] Setting key dfa0d7c1f1a03c4970881d87619c52e3
I20260620 08:08:39.300034 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd: Bootstrap starting.
I20260620 08:08:39.301072 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:39.298953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:43549 (local address 127.0.0.1:32945)
0620 08:08:39.299154 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:39.299163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:39.299205 (+ 42us) server_negotiation.cc:407] Connection header received
0620 08:08:39.299761 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:39.299770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:39.299812 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:39.299967 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:39.300634 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:39.300637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:39.300638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:39.300643 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:39.300688 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:39.300704 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:39.300802 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:39.300836 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 08:08:39.300883 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260620 08:08:39.301146 21260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:39.301452 21260 log.cc:824] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:39.302412 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd: No bootstrap required, opened a new log
I20260620 08:08:39.302675 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:39.304736 21260 raft_consensus.cc:348] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 08:08:39.304929 21260 raft_consensus.cc:374] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:39.304977 21260 raft_consensus.cc:729] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05b27ebe09b4b1089d4653db6c5ffcd, State: Initialized, Role: FOLLOWER
I20260620 08:08:39.305150 21260 consensus_queue.cc:260] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 08:08:39.305260 21260 raft_consensus.cc:388] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:39.305306 21260 raft_consensus.cc:482] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:39.305356 21260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:39.305826 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:39.304186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:42481 (local address 127.0.0.1:32945)
0620 08:08:39.304255 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:39.304257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:39.304449 (+ 192us) server_negotiation.cc:407] Connection header received
0620 08:08:39.304476 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:39.304477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:39.304483 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:39.304552 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:39.305435 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:39.305437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:39.305437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:39.305442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:39.305459 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:39.305468 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:39.305542 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:39.305717 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:08:39.305738 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:08:39.306200 21260 raft_consensus.cc:504] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 08:08:39.306313 21260 leader_election.cc:302] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d05b27ebe09b4b1089d4653db6c5ffcd; no voters:
I20260620 08:08:39.306483 21260 leader_election.cc:288] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:39.306509 21270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:39.306650 21270 raft_consensus.cc:686] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [term 1 LEADER]: Becoming Leader. State: Replica: d05b27ebe09b4b1089d4653db6c5ffcd, State: Running, Role: LEADER
I20260620 08:08:39.306754 21260 sys_catalog.cc:564] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:39.306883 21270 consensus_queue.cc:237] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260620 08:08:39.307228 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } } }
I20260620 08:08:39.307272 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d05b27ebe09b4b1089d4653db6c5ffcd. Latest consensus state: current_term: 1 leader_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05b27ebe09b4b1089d4653db6c5ffcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } } }
I20260620 08:08:39.307386 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:39.307376 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:39.307646 21279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:39.308188 21279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:39.310333 21279 catalog_manager.cc:1347] Generated new cluster ID: 0ac50c1fdc2e4e95960d079d325a3a7b
I20260620 08:08:39.310412 21279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:39.326707 21279 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:39.326925 21279 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:39.333415 21279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d05b27ebe09b4b1089d4653db6c5ffcd: Generated new TSK 0
I20260620 08:08:39.333691 21279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:39.343595 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21192
2026-06-20T08:08:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:39 dist-test-slave-c4w8 krb5kdc[21291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:39 dist-test-slave-c4w8 krb5kdc[21291](info): set up 2 sockets
Jun 20 08:08:39 dist-test-slave-c4w8 krb5kdc[21291](info): commencing operation
krb5kdc: starting...
W20260620 08:08:40.414171 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:40 dist-test-slave-c4w8 krb5kdc[21291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:40.540882 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:35173
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:35173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:40.626593 21304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:40.626891 21304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:40.626937 21304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:40.628147 21304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:40.628225 21304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:40.628249 21304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:40.628278 21304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:40.628298 21304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:40.629873 21304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:35173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:35173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:40.630172 21304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:40.630419 21304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:40.635648 21312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:40.635627 21310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:40.635918 21304 server_base.cc:1034] running on GCE node
W20260620 08:08:40.635627 21309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:40.636237 21304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:40.636545 21304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:40.637709 21304 hybrid_clock.cc:648] HybridClock initialized: now 1781942920637690 us; error 41 us; skew 500 ppm
Jun 20 08:08:40 dist-test-slave-c4w8 krb5kdc[21291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942920, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:40.640728 21304 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:40.641215 21304 webserver.cc:460] Webserver started at http://127.31.115.254:45351/ using document root <none> and password file <none>
I20260620 08:08:40.641485 21304 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:40.641541 21304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:40.641790 21304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:40.643335 21304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "daba4a791a5c4a5dab039bba918fa263"
format_stamp: "Formatted at 2026-06-20 08:08:40 on dist-test-slave-c4w8"
server_key: "a03b337ccd971de15a78b9714b48c467"
server_key_iv: "d74fe04804f5c57985754760af8b5eec"
server_key_version: "encryptionkey@0"
I20260620 08:08:40.643759 21304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "daba4a791a5c4a5dab039bba918fa263"
format_stamp: "Formatted at 2026-06-20 08:08:40 on dist-test-slave-c4w8"
server_key: "a03b337ccd971de15a78b9714b48c467"
server_key_iv: "d74fe04804f5c57985754760af8b5eec"
server_key_version: "encryptionkey@0"
I20260620 08:08:40.645551 21304 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 08:08:40.646602 21319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:40.647014 21304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:08:40.647146 21304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "daba4a791a5c4a5dab039bba918fa263"
format_stamp: "Formatted at 2026-06-20 08:08:40 on dist-test-slave-c4w8"
server_key: "a03b337ccd971de15a78b9714b48c467"
server_key_iv: "d74fe04804f5c57985754760af8b5eec"
server_key_version: "encryptionkey@0"
I20260620 08:08:40.647300 21304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:40.660071 21304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:40.663442 21304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:40.663635 21304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:40.669191 21304 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:35173
I20260620 08:08:40.669235 21371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:35173 every 8 connection(s)
I20260620 08:08:40.669680 21304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:40.671051 21372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:40.674090 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263: Bootstrap starting.
I20260620 08:08:40.675001 21372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:40.675315 21372 log.cc:824] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:40.676345 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263: No bootstrap required, opened a new log
I20260620 08:08:40.676743 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21304
I20260620 08:08:40.677067 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:40.677338 32207 external_mini_cluster.cc:1428] Setting key 8a111956e7bd37cb7052935b6162ee4d
I20260620 08:08:40.679920 21372 raft_consensus.cc:348] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba4a791a5c4a5dab039bba918fa263" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35173 } }
I20260620 08:08:40.680181 21372 raft_consensus.cc:374] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:40.680231 21372 raft_consensus.cc:729] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daba4a791a5c4a5dab039bba918fa263, State: Initialized, Role: FOLLOWER
I20260620 08:08:40.680532 21372 consensus_queue.cc:260] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba4a791a5c4a5dab039bba918fa263" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35173 } }
I20260620 08:08:40.680665 21372 raft_consensus.cc:388] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:40.680709 21372 raft_consensus.cc:482] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:40.680758 21372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:40.682185 21372 raft_consensus.cc:504] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba4a791a5c4a5dab039bba918fa263" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35173 } }
I20260620 08:08:40.682395 21372 leader_election.cc:302] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daba4a791a5c4a5dab039bba918fa263; no voters:
I20260620 08:08:40.682634 21372 leader_election.cc:288] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:40.682683 21377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:40.682818 21377 raft_consensus.cc:686] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [term 1 LEADER]: Becoming Leader. State: Replica: daba4a791a5c4a5dab039bba918fa263, State: Running, Role: LEADER
I20260620 08:08:40.682969 21372 sys_catalog.cc:564] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:40.682996 21377 consensus_queue.cc:237] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba4a791a5c4a5dab039bba918fa263" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35173 } }
I20260620 08:08:40.683400 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daba4a791a5c4a5dab039bba918fa263. Latest consensus state: current_term: 1 leader_uuid: "daba4a791a5c4a5dab039bba918fa263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba4a791a5c4a5dab039bba918fa263" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35173 } } }
I20260620 08:08:40.683662 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:40.684181 21386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:40.683579 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daba4a791a5c4a5dab039bba918fa263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba4a791a5c4a5dab039bba918fa263" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 35173 } } }
I20260620 08:08:40.684408 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:40.684722 21386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:08:40 dist-test-slave-c4w8 krb5kdc[21291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:40.688233 21386 catalog_manager.cc:1347] Generated new cluster ID: 4a9e0ff8d3d14343b27297393b05dfa6
I20260620 08:08:40.688293 21386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:40.692281 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:40.677988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33747 (local address 127.31.115.254:35173)
0620 08:08:40.678208 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:40.678217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:40.678246 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:08:40.679805 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:40.679815 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:40.679891 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:40.680046 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:40.681456 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:40.682656 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:40.683942 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:40.684644 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:40.684756 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:40.688336 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:40.688340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:40.688341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:40.688354 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:40.690488 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:40.690994 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:40.690998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:40.690998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:40.691099 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:40.691348 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:40.691349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:40.691349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:40.691637 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:40.691852 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:40.691958 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 08:08:40.692067 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":118,"threads_started":1}
I20260620 08:08:40.693315 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:40.708432 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:40.695003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43311 (local address 127.31.115.254:35173)
0620 08:08:40.695044 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:40.695046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:40.701897 (+ 6851us) server_negotiation.cc:407] Connection header received
0620 08:08:40.702010 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:40.702012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:40.702025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:40.702058 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:40.702770 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:40.703323 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:40.704050 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:40.704290 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:40.704381 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:40.705746 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:40.705748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:40.705748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:40.705756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:40.707203 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:40.707612 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:40.707613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:40.707614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:40.707671 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:40.707900 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:40.707901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:40.707902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:40.708018 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:40.708132 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:40.708249 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:08:40.708315 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:08:40.711582 21386 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:40.712163 21386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:40.718194 21386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daba4a791a5c4a5dab039bba918fa263: Generated new TSK 0
I20260620 08:08:40.718443 21386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:40.722965 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21304
W20260620 08:08:40.728097 21289 connection.cc:535] client connection to 127.31.115.254:35173 recv error: Network error: failed to read from TLS socket (remote: 127.31.115.254:35173): Connection reset by peer (error 104)
2026-06-20T08:08:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:40 dist-test-slave-c4w8 krb5kdc[21403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:40 dist-test-slave-c4w8 krb5kdc[21403](info): set up 2 sockets
Jun 20 08:08:40 dist-test-slave-c4w8 krb5kdc[21403](info): commencing operation
krb5kdc: starting...
W20260620 08:08:41.778395 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:41 dist-test-slave-c4w8 krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:41.908366 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:43737
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:43737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:41.993167 21416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:41.993422 21416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:41.993464 21416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:41.994758 21416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:41.994853 21416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:41.994881 21416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:41.994912 21416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:41.994943 21416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:41.996539 21416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:43737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:41.996858 21416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:41.997109 21416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:42.002476 21421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:42.002480 21424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:42.002558 21416 server_base.cc:1034] running on GCE node
W20260620 08:08:42.002476 21422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:42.003046 21416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:42.003384 21416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:42.004628 21416 hybrid_clock.cc:648] HybridClock initialized: now 1781942922004592 us; error 54 us; skew 500 ppm
Jun 20 08:08:42 dist-test-slave-c4w8 krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942922, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:42.011585 21416 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:42.012197 21416 webserver.cc:460] Webserver started at http://127.31.115.254:45693/ using document root <none> and password file <none>
I20260620 08:08:42.012491 21416 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:42.012574 21416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:42.012743 21416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:42.014175 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "beab1c45ce4d48fb96ccd4914bd8f49e"
format_stamp: "Formatted at 2026-06-20 08:08:42 on dist-test-slave-c4w8"
server_key: "189171fd6f34073607fd0696e7e3ee16"
server_key_iv: "2a2083e240eac6f9a53db818c639a87c"
server_key_version: "encryptionkey@0"
I20260620 08:08:42.014580 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "beab1c45ce4d48fb96ccd4914bd8f49e"
format_stamp: "Formatted at 2026-06-20 08:08:42 on dist-test-slave-c4w8"
server_key: "189171fd6f34073607fd0696e7e3ee16"
server_key_iv: "2a2083e240eac6f9a53db818c639a87c"
server_key_version: "encryptionkey@0"
I20260620 08:08:42.016153 21416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:42.017140 21431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:42.017518 21416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:42.017678 21416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "beab1c45ce4d48fb96ccd4914bd8f49e"
format_stamp: "Formatted at 2026-06-20 08:08:42 on dist-test-slave-c4w8"
server_key: "189171fd6f34073607fd0696e7e3ee16"
server_key_iv: "2a2083e240eac6f9a53db818c639a87c"
server_key_version: "encryptionkey@0"
I20260620 08:08:42.017768 21416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:42.049189 21416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:42.051756 21416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:42.051978 21416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:42.056594 21416 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:43737
I20260620 08:08:42.056656 21483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:43737 every 8 connection(s)
I20260620 08:08:42.057065 21416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:42.058141 21484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:42.060611 21484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e: Bootstrap starting.
I20260620 08:08:42.061203 21484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:42.061491 21484 log.cc:824] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:42.062309 21484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e: No bootstrap required, opened a new log
I20260620 08:08:42.064102 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21416
I20260620 08:08:42.064230 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:42.064085 21484 raft_consensus.cc:348] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43737 } }
I20260620 08:08:42.064273 21484 raft_consensus.cc:374] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:42.064320 21484 raft_consensus.cc:729] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beab1c45ce4d48fb96ccd4914bd8f49e, State: Initialized, Role: FOLLOWER
I20260620 08:08:42.064432 32207 external_mini_cluster.cc:1428] Setting key 32bb5bd7451e2d1c2dd72cbccdc9c43c
I20260620 08:08:42.064479 21484 consensus_queue.cc:260] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43737 } }
I20260620 08:08:42.064567 21484 raft_consensus.cc:388] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:42.064616 21484 raft_consensus.cc:482] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:42.064672 21484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:42.065840 21484 raft_consensus.cc:504] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43737 } }
I20260620 08:08:42.065950 21484 leader_election.cc:302] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beab1c45ce4d48fb96ccd4914bd8f49e; no voters:
I20260620 08:08:42.066133 21484 leader_election.cc:288] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:42.066155 21489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:42.066274 21489 raft_consensus.cc:686] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [term 1 LEADER]: Becoming Leader. State: Replica: beab1c45ce4d48fb96ccd4914bd8f49e, State: Running, Role: LEADER
I20260620 08:08:42.066418 21484 sys_catalog.cc:564] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:42.066447 21489 consensus_queue.cc:237] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43737 } }
I20260620 08:08:42.066926 21491 sys_catalog.cc:455] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader beab1c45ce4d48fb96ccd4914bd8f49e. Latest consensus state: current_term: 1 leader_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43737 } } }
I20260620 08:08:42.067019 21491 sys_catalog.cc:458] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:42.067230 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beab1c45ce4d48fb96ccd4914bd8f49e" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 43737 } } }
I20260620 08:08:42.067341 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:42.067509 21497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:42.068094 21497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:42.069926 21497 catalog_manager.cc:1347] Generated new cluster ID: f987d412d5cf40cc9397a57994f049d1
I20260620 08:08:42.069983 21497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:42.071311 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:42.065130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38765 (local address 127.31.115.254:43737)
0620 08:08:42.065430 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:42.065442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:42.065486 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:08:42.065629 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:42.065636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:42.065707 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:42.065856 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:42.066156 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:42.067683 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:42.068962 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:42.069531 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:42.069679 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:42.070549 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:42.070553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:42.070554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:42.070784 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:42.070826 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:42.070844 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:42.070940 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:42.071043 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:08:42.071074 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":164,"threads_started":1}
I20260620 08:08:42.072427 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:42.078487 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:42.074098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54535 (local address 127.31.115.254:43737)
0620 08:08:42.074157 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:42.074159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:42.074342 (+ 183us) server_negotiation.cc:407] Connection header received
0620 08:08:42.074451 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:42.074452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:42.074477 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:42.074538 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:42.074700 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:42.075600 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:42.076662 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:42.077051 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:42.077145 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:42.078097 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:42.078099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:42.078100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:42.078106 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:42.078124 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:42.078129 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:42.078209 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:42.078371 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 08:08:42.078389 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:08:42.098830 21497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:42.099424 21497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:42.107029 21497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beab1c45ce4d48fb96ccd4914bd8f49e: Generated new TSK 0
I20260620 08:08:42.107282 21497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:42.118585 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21416
W20260620 08:08:42.124225 21401 connection.cc:535] client connection to 127.31.115.254:43737 recv error: Network error: failed to read from TLS socket (remote: 127.31.115.254:43737): Connection reset by peer (error 104)
2026-06-20T08:08:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:42 dist-test-slave-c4w8 krb5kdc[21515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:42 dist-test-slave-c4w8 krb5kdc[21515](info): set up 2 sockets
Jun 20 08:08:42 dist-test-slave-c4w8 krb5kdc[21515](info): commencing operation
krb5kdc: starting...
W20260620 08:08:43.182271 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:43 dist-test-slave-c4w8 krb5kdc[21515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:43.308737 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:46839
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:46839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:43.395560 21528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:43.395802 21528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:43.395828 21528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:43.396996 21528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:43.397038 21528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:43.397053 21528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:43.397066 21528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:43.397078 21528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:43.398624 21528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:46839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:43.398854 21528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:43.399039 21528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:43.404402 21536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:43.404512 21533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:43.404402 21534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:43.404805 21528 server_base.cc:1034] running on GCE node
I20260620 08:08:43.405058 21528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:43.405361 21528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:43.406535 21528 hybrid_clock.cc:648] HybridClock initialized: now 1781942923406513 us; error 35 us; skew 500 ppm
Jun 20 08:08:43 dist-test-slave-c4w8 krb5kdc[21515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942923, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:43.409405 21528 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:43.409967 21528 webserver.cc:460] Webserver started at http://127.31.115.254:34921/ using document root <none> and password file <none>
I20260620 08:08:43.410190 21528 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:43.410241 21528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:43.410341 21528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:43.411522 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "ccea1b7eec8c4e91992c6712643928f3"
format_stamp: "Formatted at 2026-06-20 08:08:43 on dist-test-slave-c4w8"
server_key: "e0443e4b14b2d3b8e9a7e44cdf3cc75c"
server_key_iv: "6acde60bbee1a62a5f3b7225d7d07b85"
server_key_version: "encryptionkey@0"
I20260620 08:08:43.411880 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "ccea1b7eec8c4e91992c6712643928f3"
format_stamp: "Formatted at 2026-06-20 08:08:43 on dist-test-slave-c4w8"
server_key: "e0443e4b14b2d3b8e9a7e44cdf3cc75c"
server_key_iv: "6acde60bbee1a62a5f3b7225d7d07b85"
server_key_version: "encryptionkey@0"
I20260620 08:08:43.413411 21528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:08:43.414405 21543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:43.414722 21528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:43.414846 21528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "ccea1b7eec8c4e91992c6712643928f3"
format_stamp: "Formatted at 2026-06-20 08:08:43 on dist-test-slave-c4w8"
server_key: "e0443e4b14b2d3b8e9a7e44cdf3cc75c"
server_key_iv: "6acde60bbee1a62a5f3b7225d7d07b85"
server_key_version: "encryptionkey@0"
I20260620 08:08:43.414953 21528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:43.418583 21528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:43.420857 21528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:43.421051 21528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:43.425537 21528 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:46839
I20260620 08:08:43.425690 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:46839 every 8 connection(s)
I20260620 08:08:43.426005 21528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:43.427064 21596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:43.429414 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3: Bootstrap starting.
I20260620 08:08:43.430087 21596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:43.430418 21596 log.cc:824] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:43.431208 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3: No bootstrap required, opened a new log
I20260620 08:08:43.432917 21596 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccea1b7eec8c4e91992c6712643928f3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46839 } }
I20260620 08:08:43.433060 21596 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:43.433082 21596 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccea1b7eec8c4e91992c6712643928f3, State: Initialized, Role: FOLLOWER
I20260620 08:08:43.433252 21596 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccea1b7eec8c4e91992c6712643928f3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46839 } }
I20260620 08:08:43.433347 21596 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:43.433393 21596 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:43.433444 21596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:43.434322 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21528
I20260620 08:08:43.434494 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:43.434571 21596 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccea1b7eec8c4e91992c6712643928f3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46839 } }
I20260620 08:08:43.434705 32207 external_mini_cluster.cc:1428] Setting key ca6e14613e98f992c38dce66f516ed76
I20260620 08:08:43.434712 21596 leader_election.cc:302] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccea1b7eec8c4e91992c6712643928f3; no voters:
I20260620 08:08:43.434911 21596 leader_election.cc:288] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:43.434937 21599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:43.435060 21599 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [term 1 LEADER]: Becoming Leader. State: Replica: ccea1b7eec8c4e91992c6712643928f3, State: Running, Role: LEADER
I20260620 08:08:43.435212 21596 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:43.435226 21599 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccea1b7eec8c4e91992c6712643928f3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46839 } }
I20260620 08:08:43.435657 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccea1b7eec8c4e91992c6712643928f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccea1b7eec8c4e91992c6712643928f3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46839 } } }
I20260620 08:08:43.435899 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:43.435823 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccea1b7eec8c4e91992c6712643928f3. Latest consensus state: current_term: 1 leader_uuid: "ccea1b7eec8c4e91992c6712643928f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccea1b7eec8c4e91992c6712643928f3" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46839 } } }
I20260620 08:08:43.436128 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:43.436282 21610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:43.436843 21610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:43.438016 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:43.435415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35933 (local address 127.31.115.254:46839)
0620 08:08:43.436289 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:43.436300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:43.436335 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:08:43.436475 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:43.436482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:43.436516 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:43.436672 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:43.437448 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:43.437452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:43.437452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:43.437457 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:43.437499 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:43.437513 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:43.437584 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:43.437778 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 08:08:43.437827 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":70,"threads_started":1}
I20260620 08:08:43.438791 21610 catalog_manager.cc:1347] Generated new cluster ID: 20914477c37340989282facaa65fff27
I20260620 08:08:43.438853 21610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:43.438969 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:43.441488 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:43.440330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55825 (local address 127.31.115.254:46839)
0620 08:08:43.440389 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:43.440391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:43.440513 (+ 122us) server_negotiation.cc:407] Connection header received
0620 08:08:43.440589 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:43.440590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:43.440596 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:43.440651 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:43.441258 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:43.441260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:43.441260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:43.441264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:43.441279 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:43.441283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:43.441340 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:43.441403 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 08:08:43.441419 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:08:43.445864 21610 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:43.446036 21610 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:43.457036 21610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccea1b7eec8c4e91992c6712643928f3: Generated new TSK 0
I20260620 08:08:43.457257 21610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:43.466164 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21528
2026-06-20T08:08:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:43 dist-test-slave-c4w8 krb5kdc[21627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:43 dist-test-slave-c4w8 krb5kdc[21627](info): set up 2 sockets
Jun 20 08:08:43 dist-test-slave-c4w8 krb5kdc[21627](info): commencing operation
krb5kdc: starting...
W20260620 08:08:44.510637 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:44 dist-test-slave-c4w8 krb5kdc[21627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:44.649995 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33417
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:33417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:44.736078 21640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:44.736305 21640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:44.736331 21640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:44.737493 21640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:44.737530 21640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:44.737545 21640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:44.737558 21640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:44.737571 21640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:44.739190 21640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:33417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:44.739419 21640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:44.739599 21640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:44.745229 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:44.745244 21640 server_base.cc:1034] running on GCE node
W20260620 08:08:44.745275 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:44.745330 21648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:44.745779 21640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:44.746131 21640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:44.747311 21640 hybrid_clock.cc:648] HybridClock initialized: now 1781942924747301 us; error 52 us; skew 500 ppm
Jun 20 08:08:44 dist-test-slave-c4w8 krb5kdc[21627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942924, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:44.750174 21640 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:44.750653 21640 webserver.cc:460] Webserver started at http://127.31.115.254:40257/ using document root <none> and password file <none>
I20260620 08:08:44.750903 21640 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:44.750950 21640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:44.751093 21640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:44.752221 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "5fced45d757a4231af1c7493fc5d54f8"
format_stamp: "Formatted at 2026-06-20 08:08:44 on dist-test-slave-c4w8"
server_key: "d2123ec8cac9570d39325278d9488070"
server_key_iv: "0b9b0cb9ed9e7dcf18e2c36b0efc4d8f"
server_key_version: "encryptionkey@0"
I20260620 08:08:44.752604 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "5fced45d757a4231af1c7493fc5d54f8"
format_stamp: "Formatted at 2026-06-20 08:08:44 on dist-test-slave-c4w8"
server_key: "d2123ec8cac9570d39325278d9488070"
server_key_iv: "0b9b0cb9ed9e7dcf18e2c36b0efc4d8f"
server_key_version: "encryptionkey@0"
I20260620 08:08:44.754158 21640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:44.755106 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:44.755446 21640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:44.755563 21640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "5fced45d757a4231af1c7493fc5d54f8"
format_stamp: "Formatted at 2026-06-20 08:08:44 on dist-test-slave-c4w8"
server_key: "d2123ec8cac9570d39325278d9488070"
server_key_iv: "0b9b0cb9ed9e7dcf18e2c36b0efc4d8f"
server_key_version: "encryptionkey@0"
I20260620 08:08:44.755667 21640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:44.776223 21640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:44.778753 21640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:44.778939 21640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:44.783170 21640 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:33417
I20260620 08:08:44.783214 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:33417 every 8 connection(s)
I20260620 08:08:44.783583 21640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:44.784583 21708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:44.786077 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21640
I20260620 08:08:44.786201 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:44.786401 32207 external_mini_cluster.cc:1428] Setting key f83814e2e0e37d2713187852f362aa5a
I20260620 08:08:44.787585 21708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8: Bootstrap starting.
I20260620 08:08:44.788204 21708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:44.788539 21708 log.cc:824] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:44.789249 21708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8: No bootstrap required, opened a new log
I20260620 08:08:44.791616 21708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fced45d757a4231af1c7493fc5d54f8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33417 } }
I20260620 08:08:44.791819 21708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:44.791862 21708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fced45d757a4231af1c7493fc5d54f8, State: Initialized, Role: FOLLOWER
I20260620 08:08:44.792006 21708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fced45d757a4231af1c7493fc5d54f8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33417 } }
I20260620 08:08:44.792075 21708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:44.792096 21708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:44.792173 21708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:44.793033 21708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fced45d757a4231af1c7493fc5d54f8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33417 } }
I20260620 08:08:44.793154 21708 leader_election.cc:302] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fced45d757a4231af1c7493fc5d54f8; no voters:
I20260620 08:08:44.793339 21708 leader_election.cc:288] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:44.793378 21713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:44.793598 21713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [term 1 LEADER]: Becoming Leader. State: Replica: 5fced45d757a4231af1c7493fc5d54f8, State: Running, Role: LEADER
I20260620 08:08:44.793778 21713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fced45d757a4231af1c7493fc5d54f8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33417 } }
I20260620 08:08:44.793668 21708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 08:08:44 dist-test-slave-c4w8 krb5kdc[21627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:44.794203 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fced45d757a4231af1c7493fc5d54f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fced45d757a4231af1c7493fc5d54f8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33417 } } }
I20260620 08:08:44.794435 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:44.794368 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fced45d757a4231af1c7493fc5d54f8. Latest consensus state: current_term: 1 leader_uuid: "5fced45d757a4231af1c7493fc5d54f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fced45d757a4231af1c7493fc5d54f8" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 33417 } } }
I20260620 08:08:44.794703 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:44.795008 21720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:44.795603 21720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:44.797435 21720 catalog_manager.cc:1347] Generated new cluster ID: 2e08091e23d243f3a5af2d62dd39dc3a
I20260620 08:08:44.797492 21720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:44.798612 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:44.787214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56825 (local address 127.31.115.254:33417)
0620 08:08:44.787473 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:44.787481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:44.787508 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:08:44.787995 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:44.788002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:44.788050 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:44.788206 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:44.788884 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:44.789935 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:44.791052 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:44.791510 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:44.791603 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:44.794724 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:44.794729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:44.794729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:44.794745 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:44.796834 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:44.797273 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:44.797277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:44.797277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:44.797390 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:44.797602 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:44.797624 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:44.797625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:44.797949 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:44.798097 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:44.798232 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 08:08:44.798390 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260620 08:08:44.803027 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:44.812353 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:44.804611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34227 (local address 127.31.115.254:33417)
0620 08:08:44.804654 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:44.804656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:44.804730 (+ 74us) server_negotiation.cc:407] Connection header received
0620 08:08:44.804795 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:44.804796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:44.804807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:44.804885 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:44.805570 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:44.806374 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:44.807474 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:44.807807 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:44.807893 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:44.809357 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:44.809359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:44.809359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:44.809367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:44.811125 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:44.811519 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:44.811521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:44.811521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:44.811589 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:44.811783 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:44.811784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:44.811784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:44.811930 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:44.812030 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:44.812156 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 08:08:44.812245 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:44.836494 21720 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:44.837110 21720 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:44.840129 21720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fced45d757a4231af1c7493fc5d54f8: Generated new TSK 0
I20260620 08:08:44.840353 21720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:44.850015 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21640
2026-06-20T08:08:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:44 dist-test-slave-c4w8 krb5kdc[21739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:44 dist-test-slave-c4w8 krb5kdc[21739](info): set up 2 sockets
Jun 20 08:08:44 dist-test-slave-c4w8 krb5kdc[21739](info): commencing operation
krb5kdc: starting...
W20260620 08:08:45.926609 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:46 dist-test-slave-c4w8 krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:46.073702 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38453
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:46.162169 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:46.162426 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:46.162469 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:46.163733 21752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:46.163816 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:46.163844 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:46.163887 21752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:46.163916 21752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:46.165473 21752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:38453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:46.165807 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:46.166054 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:46.171145 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:46.171145 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:46.171228 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:46.171461 21752 server_base.cc:1034] running on GCE node
I20260620 08:08:46.171777 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:46.172056 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:46.173228 21752 hybrid_clock.cc:648] HybridClock initialized: now 1781942926173212 us; error 30 us; skew 500 ppm
Jun 20 08:08:46 dist-test-slave-c4w8 krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942926, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:46.176003 21752 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:46.176501 21752 webserver.cc:460] Webserver started at http://127.31.115.254:43353/ using document root <none> and password file <none>
I20260620 08:08:46.176697 21752 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:46.176738 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:46.176831 21752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:46.177999 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "151f84f0bd584b5d924c3904c0711fde"
format_stamp: "Formatted at 2026-06-20 08:08:46 on dist-test-slave-c4w8"
server_key: "a781203679bedd7655a20c9a7042520b"
server_key_iv: "231cdaccd3353c7f930f66a9f9522f4d"
server_key_version: "encryptionkey@0"
I20260620 08:08:46.178305 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "151f84f0bd584b5d924c3904c0711fde"
format_stamp: "Formatted at 2026-06-20 08:08:46 on dist-test-slave-c4w8"
server_key: "a781203679bedd7655a20c9a7042520b"
server_key_iv: "231cdaccd3353c7f930f66a9f9522f4d"
server_key_version: "encryptionkey@0"
I20260620 08:08:46.179773 21752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:46.180670 21767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:46.180964 21752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:46.181094 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "151f84f0bd584b5d924c3904c0711fde"
format_stamp: "Formatted at 2026-06-20 08:08:46 on dist-test-slave-c4w8"
server_key: "a781203679bedd7655a20c9a7042520b"
server_key_iv: "231cdaccd3353c7f930f66a9f9522f4d"
server_key_version: "encryptionkey@0"
I20260620 08:08:46.181188 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:46.191432 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:46.193727 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:46.193898 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:46.198242 21752 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:38453
I20260620 08:08:46.198338 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:38453 every 8 connection(s)
I20260620 08:08:46.198663 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:46.199585 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:46.199903 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21752
I20260620 08:08:46.200029 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:46.200244 32207 external_mini_cluster.cc:1428] Setting key 8dab0a1c5394f75c7f8826b05a687821
I20260620 08:08:46.203051 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde: Bootstrap starting.
I20260620 08:08:46.203856 21820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:46.204156 21820 log.cc:824] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:46.204967 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde: No bootstrap required, opened a new log
I20260620 08:08:46.207021 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:46.200807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51205 (local address 127.31.115.254:38453)
0620 08:08:46.201045 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:46.201056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:46.201094 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:08:46.201627 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:46.201637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:46.201711 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:46.201872 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:46.202119 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:46.203535 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:46.204471 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:46.205035 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:46.205150 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:46.206004 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:46.206009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:46.206009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:46.206221 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:46.206266 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:46.206282 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:46.206357 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:46.206508 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:08:46.206581 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
I20260620 08:08:46.207273 21820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151f84f0bd584b5d924c3904c0711fde" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38453 } }
I20260620 08:08:46.207468 21820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:46.207510 21820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151f84f0bd584b5d924c3904c0711fde, State: Initialized, Role: FOLLOWER
I20260620 08:08:46.207629 21820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151f84f0bd584b5d924c3904c0711fde" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38453 } }
I20260620 08:08:46.207731 21820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:46.207773 21820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:46.207823 21820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:46.208268 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:46.208649 21820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151f84f0bd584b5d924c3904c0711fde" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38453 } }
I20260620 08:08:46.208777 21820 leader_election.cc:302] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 151f84f0bd584b5d924c3904c0711fde; no voters:
I20260620 08:08:46.209000 21820 leader_election.cc:288] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:46.209033 21825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:46.209143 21825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [term 1 LEADER]: Becoming Leader. State: Replica: 151f84f0bd584b5d924c3904c0711fde, State: Running, Role: LEADER
I20260620 08:08:46.209277 21825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151f84f0bd584b5d924c3904c0711fde" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38453 } }
I20260620 08:08:46.209278 21820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:46.209981 21827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "151f84f0bd584b5d924c3904c0711fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151f84f0bd584b5d924c3904c0711fde" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38453 } } }
I20260620 08:08:46.210026 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 151f84f0bd584b5d924c3904c0711fde. Latest consensus state: current_term: 1 leader_uuid: "151f84f0bd584b5d924c3904c0711fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151f84f0bd584b5d924c3904c0711fde" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 38453 } } }
I20260620 08:08:46.210150 21827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:46.210258 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:46.210750 21835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:46.211405 21835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:46.213593 21835 catalog_manager.cc:1347] Generated new cluster ID: 6d466a0deac44e549c87a44b54979b96
I20260620 08:08:46.213685 21835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:46.218298 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:46.209803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51703 (local address 127.31.115.254:38453)
0620 08:08:46.209905 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:46.209907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:46.210488 (+ 581us) server_negotiation.cc:407] Connection header received
0620 08:08:46.210566 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:46.210567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:46.210579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:46.210630 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:46.210821 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:46.211667 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:46.212781 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:46.213072 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:46.213153 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:46.218006 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:46.218007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:46.218008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:46.218015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:46.218036 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:46.218042 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:46.218103 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:46.218178 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:08:46.218210 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 08:08:46.219879 21835 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:46.220484 21835 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:46.228124 21835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 151f84f0bd584b5d924c3904c0711fde: Generated new TSK 0
I20260620 08:08:46.228421 21835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:46.235297 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21752
2026-06-20T08:08:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:46 dist-test-slave-c4w8 krb5kdc[21851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:46 dist-test-slave-c4w8 krb5kdc[21851](info): set up 2 sockets
Jun 20 08:08:46 dist-test-slave-c4w8 krb5kdc[21851](info): commencing operation
krb5kdc: starting...
W20260620 08:08:47.315506 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:47 dist-test-slave-c4w8 krb5kdc[21851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:47.468055 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40355
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:47.557469 21864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:47.557780 21864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:47.557813 21864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:47.559219 21864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:47.559269 21864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:47.559283 21864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:47.559296 21864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:47.559392 21864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:47.562693 21864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:47.563069 21864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:47.563428 21864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:47.569509 21869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:47.569497 21870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:47.569497 21872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:47.569806 21864 server_base.cc:1034] running on GCE node
I20260620 08:08:47.570080 21864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:47.570366 21864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:47.571626 21864 hybrid_clock.cc:648] HybridClock initialized: now 1781942927571611 us; error 35 us; skew 500 ppm
Jun 20 08:08:47 dist-test-slave-c4w8 krb5kdc[21851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942927, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:47.579767 21864 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:47.580523 21864 webserver.cc:460] Webserver started at http://127.31.115.254:41705/ using document root <none> and password file <none>
I20260620 08:08:47.581008 21864 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:47.581094 21864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:47.581302 21864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:47.582528 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "8eec51d25af34b81a8bfb17c892861a2"
format_stamp: "Formatted at 2026-06-20 08:08:47 on dist-test-slave-c4w8"
server_key: "c2a02666160854718be9067a53571ec9"
server_key_iv: "e649ef7415872f12b1e73d9ab95f3da8"
server_key_version: "encryptionkey@0"
I20260620 08:08:47.582919 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "8eec51d25af34b81a8bfb17c892861a2"
format_stamp: "Formatted at 2026-06-20 08:08:47 on dist-test-slave-c4w8"
server_key: "c2a02666160854718be9067a53571ec9"
server_key_iv: "e649ef7415872f12b1e73d9ab95f3da8"
server_key_version: "encryptionkey@0"
I20260620 08:08:47.584451 21864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:08:47.585772 21879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:47.586133 21864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:47.586232 21864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "8eec51d25af34b81a8bfb17c892861a2"
format_stamp: "Formatted at 2026-06-20 08:08:47 on dist-test-slave-c4w8"
server_key: "c2a02666160854718be9067a53571ec9"
server_key_iv: "e649ef7415872f12b1e73d9ab95f3da8"
server_key_version: "encryptionkey@0"
I20260620 08:08:47.586288 21864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:47.592288 21864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:47.595008 21864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:47.595222 21864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:47.599867 21864 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:40355
I20260620 08:08:47.599895 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:40355 every 8 connection(s)
I20260620 08:08:47.600311 21864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:47.601351 21932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:47.603790 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2: Bootstrap starting.
I20260620 08:08:47.604398 21932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:47.604707 21932 log.cc:824] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:47.605424 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2: No bootstrap required, opened a new log
I20260620 08:08:47.605558 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21864
I20260620 08:08:47.605739 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:47.606017 32207 external_mini_cluster.cc:1428] Setting key e88a0c4c3c227e5ba1c32c50797d34e3
I20260620 08:08:47.607903 21932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec51d25af34b81a8bfb17c892861a2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40355 } }
I20260620 08:08:47.608094 21932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:08:47.608137 21932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eec51d25af34b81a8bfb17c892861a2, State: Initialized, Role: FOLLOWER
I20260620 08:08:47.608307 21932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec51d25af34b81a8bfb17c892861a2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40355 } }
I20260620 08:08:47.608421 21932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:08:47.608470 21932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:08:47.608523 21932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:47.609499 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:47.606735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39207 (local address 127.31.115.254:40355)
0620 08:08:47.606918 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:47.606933 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:47.606961 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:08:47.607886 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:47.607896 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:47.607930 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:47.608073 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:47.608950 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:47.608954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:47.608954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:47.608958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:47.609002 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:47.609015 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:47.609084 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:47.609184 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 08:08:47.609256 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260620 08:08:47.609726 21932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec51d25af34b81a8bfb17c892861a2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40355 } }
I20260620 08:08:47.609841 21932 leader_election.cc:302] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eec51d25af34b81a8bfb17c892861a2; no voters:
I20260620 08:08:47.610019 21932 leader_election.cc:288] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:08:47.610145 21937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:47.610278 21932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:47.610873 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:47.610790 21937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [term 1 LEADER]: Becoming Leader. State: Replica: 8eec51d25af34b81a8bfb17c892861a2, State: Running, Role: LEADER
I20260620 08:08:47.612479 21937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec51d25af34b81a8bfb17c892861a2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40355 } }
I20260620 08:08:47.613088 21939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eec51d25af34b81a8bfb17c892861a2. Latest consensus state: current_term: 1 leader_uuid: "8eec51d25af34b81a8bfb17c892861a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec51d25af34b81a8bfb17c892861a2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40355 } } }
I20260620 08:08:47.613234 21939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:47.613168 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eec51d25af34b81a8bfb17c892861a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec51d25af34b81a8bfb17c892861a2" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40355 } } }
I20260620 08:08:47.613343 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:47.613777 21956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:47.614650 21956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:47.616175 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:47.612756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44867 (local address 127.31.115.254:40355)
0620 08:08:47.612825 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:47.612828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:47.613147 (+ 319us) server_negotiation.cc:407] Connection header received
0620 08:08:47.613446 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:47.613447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:47.613454 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:47.613557 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:47.615879 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:47.615881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:47.615882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:08:47.615887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:47.615911 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:08:47.615915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:47.615944 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:47.616091 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:08:47.616110 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:08:47.616540 21956 catalog_manager.cc:1347] Generated new cluster ID: 47393af72a344f52b2eccacbdbc857b1
I20260620 08:08:47.616603 21956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:47.628341 21956 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:47.628574 21956 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:47.633715 21956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eec51d25af34b81a8bfb17c892861a2: Generated new TSK 0
I20260620 08:08:47.633924 21956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:47.639663 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21864
2026-06-20T08:08:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1405 ms)
[----------] 18 tests from AuthTokenIssuingTest (24891 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:47 dist-test-slave-c4w8 krb5kdc[21963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:47 dist-test-slave-c4w8 krb5kdc[21963](info): set up 2 sockets
Jun 20 08:08:47 dist-test-slave-c4w8 krb5kdc[21963](info): commencing operation
krb5kdc: starting...
W20260620 08:08:48.774371 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:48 dist-test-slave-c4w8 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:48.928447 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40013
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40013,127.31.115.253:46825,127.31.115.252:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:49.030287 21976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:08:49.030720 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:49.030812 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:49.030855 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:49.032466 21976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:49.032570 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:49.032599 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:49.032627 21976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:49.032660 21976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:49.034641 21976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40013,127.31.115.253:46825,127.31.115.252:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:49.034988 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:49.035249 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:49.041049 21982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:49.041074 21984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:49.041077 21981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:49.041181 21976 server_base.cc:1034] running on GCE node
I20260620 08:08:49.041579 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:49.042213 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:49.043447 21976 hybrid_clock.cc:648] HybridClock initialized: now 1781942929043395 us; error 118 us; skew 500 ppm
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:49.046764 21976 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:49.047359 21976 webserver.cc:460] Webserver started at http://127.31.115.254:35785/ using document root <none> and password file <none>
I20260620 08:08:49.047624 21976 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:49.047696 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:49.047863 21976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:49.049364 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "54f840e1cab94e89affc1488bb854bf4"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "62899bca28ef8edae30e53c8511c9d8a"
server_key_iv: "35bf1d95c85c5d2463a1fda8d6645b46"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.049907 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "54f840e1cab94e89affc1488bb854bf4"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "62899bca28ef8edae30e53c8511c9d8a"
server_key_iv: "35bf1d95c85c5d2463a1fda8d6645b46"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.051832 21976 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 08:08:49.053071 21991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:49.053406 21976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:49.053553 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "54f840e1cab94e89affc1488bb854bf4"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "62899bca28ef8edae30e53c8511c9d8a"
server_key_iv: "35bf1d95c85c5d2463a1fda8d6645b46"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.053685 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:49.075341 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:49.078208 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:49.078411 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:49.082762 21976 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:40013
I20260620 08:08:49.082808 22043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:40013 every 8 connection(s)
I20260620 08:08:49.083215 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:49.084201 22044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:49.085950 22044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.086408 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 21976
I20260620 08:08:49.086572 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:49.086810 32207 external_mini_cluster.cc:1428] Setting key 48a3b1e002c5a4f0c92479e27b36b7a0
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.253@KRBTEST.COM" created.
I20260620 08:08:49.101776 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.086829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40013 (local address 127.0.0.1:54008)
0620 08:08:49.087085 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.087141 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.087810 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.088342 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.088347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.089084 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.089444 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.089453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.090941 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.090942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.092015 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.092019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.093102 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.093103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.093176 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.093878 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.093887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.095699 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.099126 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.099133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.099136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.099375 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.099710 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.099711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.099712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.099884 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.100658 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.100672 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.100821 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.101164 (+ 343us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.101418 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":154,"threads_started":1}
I20260620 08:08:49.101783 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.087053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.31.115.254:40013)
0620 08:08:49.087210 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.087215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.087256 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:08:49.088077 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.088081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.088110 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.088248 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.089661 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.090772 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.092170 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.092973 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.093127 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.095832 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.095841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.095841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.095850 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.098920 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.099506 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.099510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.099510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.099603 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.099978 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.099979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.099980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.100519 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.100617 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.101292 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.101581 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260620 08:08:49.102774 22044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.103336 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.103088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46314)
0620 08:08:49.103216 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.103275 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 08:08:49.103571 21992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
I20260620 08:08:49.104175 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.104043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46328)
0620 08:08:49.104122 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.104154 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 08:08:49.104312 22044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:46825: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab.
Entry for principal kudu/127.31.115.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab.
I20260620 08:08:49.122102 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:46825
--webserver_interface=127.31.115.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab
--principal=kudu/127.31.115.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40013,127.31.115.253:46825,127.31.115.252:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:08:49.153699 22044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } attempt: 1
I20260620 08:08:49.154223 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.154082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46340)
0620 08:08:49.154150 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.154194 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:08:49.154598 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.154496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46354)
0620 08:08:49.154545 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.154573 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:08:49.154703 22044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:46825: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
W20260620 08:08:49.219823 22051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:08:49.220099 22051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:49.220142 22051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:49.220185 22051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:49.221539 22051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:49.221836 22051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:49.221912 22051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:49.221947 22051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:49.221982 22051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:49.222895 22044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } attempt: 2
I20260620 08:08:49.223402 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.223230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46360)
0620 08:08:49.223309 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.223362 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 08:08:49.223970 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.223856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46362)
0620 08:08:49.223911 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.223945 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:08:49.224121 22044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:46825: Network error: Client connection negotiation failed: client connection to 127.31.115.253:46825: connect: Connection refused (error 111)
I20260620 08:08:49.224335 22051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40013,127.31.115.253:46825,127.31.115.252:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab
--principal=kudu/127.31.115.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.115.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:49.224629 22051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:49.224910 22051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:49.230495 22059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:49.230459 22057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:49.230445 22056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:49.230715 22051 server_base.cc:1034] running on GCE node
I20260620 08:08:49.231058 22051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:49.231391 22051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:49.232568 22051 hybrid_clock.cc:648] HybridClock initialized: now 1781942929232545 us; error 41 us; skew 500 ppm
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:49.235567 22051 init.cc:377] Logged in from keytab as kudu/127.31.115.253@KRBTEST.COM (short username kudu)
I20260620 08:08:49.236157 22051 webserver.cc:460] Webserver started at http://127.31.115.253:37703/ using document root <none> and password file <none>
I20260620 08:08:49.236440 22051 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:49.236514 22051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:49.236614 22051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:49.237872 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data/instance:
uuid: "a984f279252d4a1d802762e2b9b7de1d"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "a4768ae46c0481bce0d0ecb773bcc356"
server_key_iv: "88fca614c13da203e54ab84bd2a47d99"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.238265 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal/instance:
uuid: "a984f279252d4a1d802762e2b9b7de1d"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "a4768ae46c0481bce0d0ecb773bcc356"
server_key_iv: "88fca614c13da203e54ab84bd2a47d99"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.239770 22051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:49.240986 22066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:49.241215 22051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:49.241308 22051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal
uuid: "a984f279252d4a1d802762e2b9b7de1d"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "a4768ae46c0481bce0d0ecb773bcc356"
server_key_iv: "88fca614c13da203e54ab84bd2a47d99"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.241379 22051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:49.250329 22051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:49.252905 22051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:49.253046 22051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:49.258239 22051 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.253:46825
I20260620 08:08:49.258285 22118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.253:46825 every 8 connection(s)
I20260620 08:08:49.258790 22051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
I20260620 08:08:49.259912 22119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:49.261834 22119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.268692 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 22051
I20260620 08:08:49.268815 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-1/wal/instance
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:49.269098 32207 external_mini_cluster.cc:1428] Setting key 8e5ca0ce462eab96cafac69d5996e97c
I20260620 08:08:49.274019 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.262710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.31.115.254:40013)
0620 08:08:49.262809 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.262811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.263036 (+ 225us) server_negotiation.cc:407] Connection header received
0620 08:08:49.263541 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.263542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.263554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.263621 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.264381 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.264895 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.266446 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.266675 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.266751 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.269651 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.269654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.269655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.269668 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.271610 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.272301 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.272303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.272303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.272492 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.272916 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.272917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.272918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.273099 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.273183 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.273696 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.273815 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 08:08:49.274040 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.262651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40013 (local address 127.0.0.1:54020)
0620 08:08:49.262839 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.262886 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.263422 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.263727 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.263731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.264032 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.264231 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.264235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.265125 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.265128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.266300 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.266304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.266869 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.266870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.267023 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.267607 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.267617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.269473 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.271720 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.271730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.271733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.272168 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.272596 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.272597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.272598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.272813 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.273204 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.273211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.273470 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.273591 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.273850 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260620 08:08:49.275341 22119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.115.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.252@KRBTEST.COM" created.
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:49.286731 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.275775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.31.115.253:46825)
0620 08:08:49.276004 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.276009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.276036 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:08:49.276056 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.276058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.276070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.276135 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.276579 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.277447 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.278780 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.279264 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.279348 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.283140 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.283155 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.283155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.283166 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.285126 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.285590 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.285594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.285594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.285720 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.286004 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.286005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.286005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.286198 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.286285 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.286502 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.286621 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":197,"thread_start_us":169,"threads_started":1}
I20260620 08:08:49.286751 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.275696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46370)
0620 08:08:49.275769 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.275783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.275851 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.276164 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.276165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.276351 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.276432 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.276434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.277587 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.277589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.278611 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.278614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.279369 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.279370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.279446 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.280182 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.280190 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.283026 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.285238 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.285239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.285240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.285486 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.285801 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.285802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.285802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.285883 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.286282 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.286283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.286339 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.286426 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.286562 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 08:08:49.287602 22119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.288024 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.287865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46906)
0620 08:08:49.287926 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.287987 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:08:49.288220 22068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
I20260620 08:08:49.288836 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.288711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46918)
0620 08:08:49.288766 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.288800 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:08:49.289042 22119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:46727: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab.
Entry for principal kudu/127.31.115.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab.
I20260620 08:08:49.302213 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:46727
--webserver_interface=127.31.115.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab
--principal=kudu/127.31.115.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:40013,127.31.115.253:46825,127.31.115.252:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:08:49.315374 22044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } attempt: 3
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:49.325330 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.315782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46376)
0620 08:08:49.315858 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.315881 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.315935 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.316268 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.316269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.316512 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.316583 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.316585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.317602 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.317653 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.318342 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.318351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.319067 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.319068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.319164 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.319878 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.319884 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.321430 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.324022 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.324023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.324024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.324273 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.324588 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.324589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.324589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.324656 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.324995 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.324996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.325048 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.325119 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.325223 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 08:08:49.325371 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.315902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.31.115.253:46825)
0620 08:08:49.315977 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.315979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.315990 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:49.316025 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.316026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.316043 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.316135 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.316678 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.317500 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.318512 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.318910 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.319012 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.321555 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.321557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.321557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.321571 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.323876 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.324405 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.324407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.324407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.324488 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.324737 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.324738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.324738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.324894 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.324972 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.325172 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.325269 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:08:49.326169 22044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.326670 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.326489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46928)
0620 08:08:49.326599 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.326640 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 08:08:49.327042 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.326941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46944)
0620 08:08:49.326994 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.327020 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:08:49.327144 22044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:46727: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
I20260620 08:08:49.338307 22119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } attempt: 1
I20260620 08:08:49.338899 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.338736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46948)
0620 08:08:49.338807 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.338859 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:08:49.339288 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.339180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46960)
0620 08:08:49.339231 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.339265 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:08:49.339396 22119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:46727: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
I20260620 08:08:49.358350 22044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } attempt: 1
I20260620 08:08:49.359092 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.358897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46966)
0620 08:08:49.359003 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.359051 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 08:08:49.359625 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.359492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46968)
0620 08:08:49.359561 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.359600 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 08:08:49.359829 22044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:46727: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
W20260620 08:08:49.395529 22126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:08:49.395774 22126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:49.395794 22126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:49.395818 22126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:49.397094 22126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:49.397281 22126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:49.397362 22126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:49.397382 22126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:49.397395 22126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:49.399448 22126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:40013,127.31.115.253:46825,127.31.115.252:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab
--principal=kudu/127.31.115.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.115.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:49.399791 22126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:49.400090 22126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:49.405462 22134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:49.405423 22132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:49.405668 22131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:49.405838 22126 server_base.cc:1034] running on GCE node
I20260620 08:08:49.406147 22126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:49.406468 22126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:49.407663 22126 hybrid_clock.cc:648] HybridClock initialized: now 1781942929407646 us; error 29 us; skew 500 ppm
I20260620 08:08:49.407590 22119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } attempt: 2
I20260620 08:08:49.408322 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.408146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46976)
0620 08:08:49.408229 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.408275 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 08:08:49.408779 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.408665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46988)
0620 08:08:49.408719 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.408751 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:08:49.409139 22119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:46727: Network error: Client connection negotiation failed: client connection to 127.31.115.252:46727: connect: Connection refused (error 111)
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:49.411371 22126 init.cc:377] Logged in from keytab as kudu/127.31.115.252@KRBTEST.COM (short username kudu)
I20260620 08:08:49.411897 22126 webserver.cc:460] Webserver started at http://127.31.115.252:38391/ using document root <none> and password file <none>
I20260620 08:08:49.412169 22126 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:49.412222 22126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:49.412356 22126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:49.413553 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data/instance:
uuid: "3f0aca6f7dc94e8da2740558046b5307"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "c7064827477a20f0a6f7ccac39d89c18"
server_key_iv: "13b0495d95b2d7f10cc0e1e0053a402e"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.414018 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal/instance:
uuid: "3f0aca6f7dc94e8da2740558046b5307"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "c7064827477a20f0a6f7ccac39d89c18"
server_key_iv: "13b0495d95b2d7f10cc0e1e0053a402e"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.415561 22126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:49.416522 22141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:49.417007 22126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:49.417135 22126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal
uuid: "3f0aca6f7dc94e8da2740558046b5307"
format_stamp: "Formatted at 2026-06-20 08:08:49 on dist-test-slave-c4w8"
server_key: "c7064827477a20f0a6f7ccac39d89c18"
server_key_iv: "13b0495d95b2d7f10cc0e1e0053a402e"
server_key_version: "encryptionkey@0"
I20260620 08:08:49.417228 22126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:49.427640 22126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:49.430415 22126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:49.430578 22126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:49.434938 22126 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.252:46727
I20260620 08:08:49.434976 22193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.252:46727 every 8 connection(s)
I20260620 08:08:49.435009 22044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } attempt: 2
I20260620 08:08:49.435832 22126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
I20260620 08:08:49.437101 22195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:49.438619 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 22126
I20260620 08:08:49.438799 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781942645567166-32207-0/minicluster-data/master-2/wal/instance
I20260620 08:08:49.438999 32207 external_mini_cluster.cc:1428] Setting key ed2c620d6d500ada8cdde68613f2b632
I20260620 08:08:49.439579 22195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:08:49.448180 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.435584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.31.115.252:46727)
0620 08:08:49.436169 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.436181 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.436223 (+ 42us) server_negotiation.cc:407] Connection header received
0620 08:08:49.437162 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.437172 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.437243 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.437419 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.437840 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.439389 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.440795 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.441462 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.441567 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.444398 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.444405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.444406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.444413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.446340 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.446756 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.446760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.446760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.446855 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.447093 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.447094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.447095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.447443 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.447563 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.447782 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.447920 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":527,"thread_start_us":492,"threads_started":1}
I20260620 08:08:49.448395 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.435352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:46996)
0620 08:08:49.435558 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.435573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.435640 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.437448 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.437450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.437668 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.437727 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.437730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.439710 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.439712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.440675 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.440679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.441528 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.441529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.441588 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.442107 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.442113 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.444296 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.446440 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.446441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.446441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.446673 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.446934 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.446935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.446936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.447022 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.447546 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.447547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.447599 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.447693 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.447834 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 08:08:49.450811 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4: Bootstrap starting.
I20260620 08:08:49.451116 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.440414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.31.115.254:40013)
0620 08:08:49.440472 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.440474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.440873 (+ 399us) server_negotiation.cc:407] Connection header received
0620 08:08:49.440931 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.440932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.440944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.440994 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.441482 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.442260 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.443277 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.443596 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.443676 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.446978 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.446980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.446980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.446990 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.449556 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.450170 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.450171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.450171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.450238 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.450493 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.450495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.450495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.450654 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.450700 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.450962 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.451042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:08:49.451515 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.440374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:40013 (local address 127.0.0.1:54026)
0620 08:08:49.440565 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.440606 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.440782 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.441016 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.441019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.441291 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.441395 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.441397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.442374 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.442375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.443149 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.443152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.443695 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.443696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.443784 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.444330 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.444342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.446859 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.449724 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.449731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.449733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.450052 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.450322 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.450323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.450323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.450401 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.450753 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.450760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.450812 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.451203 (+ 391us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.451352 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":33,"thread_start_us":76,"threads_started":1}
I20260620 08:08:49.452204 22044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:49.452171 22195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.452291 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.439443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.31.115.252:46727)
0620 08:08:49.439632 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.439634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.439821 (+ 187us) server_negotiation.cc:407] Connection header received
0620 08:08:49.439867 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.439868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.439880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.439954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.440873 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.441641 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.442607 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.443006 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.443118 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.447624 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.447626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.447626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.447635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.449997 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.450496 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.450497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.450498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.450575 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.450828 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.450829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.450829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.451003 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.451215 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.451617 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.452070 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260620 08:08:49.452755 22044 log.cc:824] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:49.454226 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4: No bootstrap required, opened a new log
I20260620 08:08:49.457532 22044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:49.458067 22044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f840e1cab94e89affc1488bb854bf4, State: Initialized, Role: FOLLOWER
I20260620 08:08:49.458225 22044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.458739 22044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:49.458933 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.459635 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:49.462129 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.452510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:46825 (local address 127.0.0.1:46378)
0620 08:08:49.452710 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.452724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.452766 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.452973 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.452974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.453161 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.453248 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.453250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.454475 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.454476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.455150 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.455154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.455589 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.455590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.455659 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.456739 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.456744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.458375 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.460616 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.460618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.460618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.461127 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.461356 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.461357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.461357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.461429 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.461772 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.461774 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.461822 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.461911 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.462027 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260620 08:08:49.462127 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.452636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.31.115.253:46825)
0620 08:08:49.452743 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.452747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.452763 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:08:49.452879 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.452881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.452895 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.452969 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.453347 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.454124 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.455246 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.455479 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.455589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.458518 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.458520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.458520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.458530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.460503 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.461195 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.461196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.461196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.461267 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.461503 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.461504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.461504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.461693 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.461768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.461952 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.462026 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:08:49.462833 22195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:49.463300 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.453126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.31.115.254:40013)
0620 08:08:49.453190 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.453192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.453458 (+ 266us) server_negotiation.cc:407] Connection header received
0620 08:08:49.453772 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.453774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.453791 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.453882 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.454898 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.455796 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.456718 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.457164 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.457305 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.459947 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.459949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.459949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.459957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.461721 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.462062 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.462064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.462064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.462123 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.462344 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.462345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.462345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.462491 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.462616 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.463104 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.463204 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:08:49.463186 21976 master_runner.cc:186] Error getting master registration for 127.31.115.253:46825: OK, instance_id { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" instance_seqno: 1781942929256689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:08:49.466497 21976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:08:49.466835 22212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:49.466945 22212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:49.471837 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.463113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:47006)
0620 08:08:49.463300 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.463311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.463357 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.463517 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.463518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.463676 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.463721 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.463723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.464911 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.464913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.465786 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.465789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.466326 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.466327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.466371 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.467109 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.467115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.468655 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.470663 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.470664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.470664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.470883 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.471093 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.471094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.471094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.471163 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.471478 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.471479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.471519 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.471590 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.471698 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260620 08:08:49.471958 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.463181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.31.115.252:46727)
0620 08:08:49.463216 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.463218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.463411 (+ 193us) server_negotiation.cc:407] Connection header received
0620 08:08:49.463428 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.463428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.463439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.463480 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.463804 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.464741 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.465872 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.466200 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.466263 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.468766 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.468767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.468768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.468775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.470568 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.470953 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.470955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.470955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.471013 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.471239 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.471240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.471240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.471390 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.471454 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.471656 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.471767 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 08:08:49.473801 22195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: Bootstrap starting.
I20260620 08:08:49.474567 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.464068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.31.115.253:46825)
0620 08:08:49.464102 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.464104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.464234 (+ 130us) server_negotiation.cc:407] Connection header received
0620 08:08:49.464254 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.464255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.464264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.464303 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.465314 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.466138 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.467718 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.467971 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.468073 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.470892 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.470894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.470894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.470904 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.473041 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.473498 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.473499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.473500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.473568 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.473853 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.473854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.473855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.474015 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.474183 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.474348 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.474440 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:08:49.474769 22195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:49.475131 22195 log.cc:824] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:49.476081 22195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: No bootstrap required, opened a new log
I20260620 08:08:49.476315 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:49.477257 22018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "54f840e1cab94e89affc1488bb854bf4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54036
I20260620 08:08:49.477385 22018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 08:08:49.477419 22018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:49.478318 22018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.478480 22195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.478641 22018 leader_election.cc:288] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [CANDIDATE]: Term 1 election: Requested vote from peers a984f279252d4a1d802762e2b9b7de1d (127.31.115.253:46825), 3f0aca6f7dc94e8da2740558046b5307 (127.31.115.252:46727)
I20260620 08:08:49.478693 22195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0aca6f7dc94e8da2740558046b5307, State: Initialized, Role: FOLLOWER
I20260620 08:08:49.478806 22195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.479156 22195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:49.479305 22168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54f840e1cab94e89affc1488bb854bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3f0aca6f7dc94e8da2740558046b5307"
I20260620 08:08:49.479453 22168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:49.480561 22168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f840e1cab94e89affc1488bb854bf4 in term 1.
I20260620 08:08:49.480850 21993 leader_election.cc:302] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0aca6f7dc94e8da2740558046b5307, 54f840e1cab94e89affc1488bb854bf4; no voters:
I20260620 08:08:49.481002 22201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:49.481170 22201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 54f840e1cab94e89affc1488bb854bf4, State: Running, Role: LEADER
I20260620 08:08:49.481298 22201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.481958 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f840e1cab94e89affc1488bb854bf4. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.482050 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:49.482266 22236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:49.482511 22236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:49.484302 22168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [term 1 FOLLOWER]: Refusing update from remote peer 54f840e1cab94e89affc1488bb854bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:49.484876 22201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:49.485183 22126 master_runner.cc:186] Error getting master registration for 127.31.115.253:46825: OK, instance_id { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" instance_seqno: 1781942929256689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:08:49.489823 22126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:08:49.491387 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.491518 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:49.491734 22237 mvcc.cc:205] Tried to move back new op lower bound from 7298838239164792832 to 7298838239156613120. Current Snapshot: MvccSnapshot[applied={T|T < 7298838239164792832}]
I20260620 08:08:49.491834 22239 mvcc.cc:205] Tried to move back new op lower bound from 7298838239164792832 to 7298838239156613120. Current Snapshot: MvccSnapshot[applied={T|T < 7298838239164792832}]
I20260620 08:08:49.492550 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.480347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.31.115.254:40013)
0620 08:08:49.480435 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.480437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.481451 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 08:08:49.481676 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.481677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.481690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.481746 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.483169 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.483932 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.485848 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.486144 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.486186 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.488625 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.488627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.488628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.488635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.490565 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.491101 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.491102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.491102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.491192 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.491465 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.491466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.491467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.491631 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.491748 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.492351 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.492443 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:08:49.493996 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.494143 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:49.494249 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f840e1cab94e89affc1488bb854bf4. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.494328 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:49.494381 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.494442 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:08:49.495141 22231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:49.495229 22231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:49.498318 22236 catalog_manager.cc:1347] Generated new cluster ID: 181b4822dca241b2a00d601d5ba26029
I20260620 08:08:49.498413 22236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:08:49.499409 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.481047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.31.115.252:46727)
0620 08:08:49.483125 (+ 2078us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.483127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.483801 (+ 674us) server_negotiation.cc:407] Connection header received
0620 08:08:49.483917 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.483918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.483930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.483969 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.484849 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.485645 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.486887 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.487190 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.487243 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.495334 (+ 8091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.495336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.495337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.495346 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.497757 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.498213 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.498214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.498214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.498273 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.498507 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.498508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.498509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.498663 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.498742 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.499206 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.499302 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2053}
I20260620 08:08:49.500337 22119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } attempt: 3
I20260620 08:08:49.501708 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.482703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.31.115.253:46825)
0620 08:08:49.482760 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.482763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.487959 (+ 5196us) server_negotiation.cc:407] Connection header received
0620 08:08:49.488097 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.488098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.488131 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.488171 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.492062 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.492829 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.495985 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.496367 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.496428 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.498121 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.498122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.498123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.498131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.500049 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.500643 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.500644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.500644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.500714 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.500926 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.500927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.500927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.501150 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.501253 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.501450 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.501545 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 08:08:49 dist-test-slave-c4w8 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:08:49.512176 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.500662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:46727 (local address 127.0.0.1:47032)
0620 08:08:49.500724 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:49.500736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:49.500796 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:49.501895 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:49.501896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:49.502065 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:49.502111 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.502113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.503189 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.503190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.503768 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:49.503770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.504247 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.504249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:49.504306 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.504858 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:49.504863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:49.506799 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:49.510252 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.510254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.510255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.510513 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.510838 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:49.510839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:49.510839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:49.510920 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:49.511399 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:49.511400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:49.511596 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 08:08:49.511696 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:08:49.511880 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:08:49.513407 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.501684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.31.115.252:46727)
0620 08:08:49.501751 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.501753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.501778 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:08:49.501797 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.501798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.501807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.501853 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.502308 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.503067 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.503860 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.504182 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.504239 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.507357 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:49.507360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:49.507360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:49.507369 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:49.510102 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.510652 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.510653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.510653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.510738 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:49.511010 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:49.511011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:49.511012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:49.511268 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:49.513014 (+ 1746us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.513090 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.513257 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:08:49.515218 22119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: Bootstrap starting.
I20260620 08:08:49.516057 22119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:49.516397 22119 log.cc:824] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:49.517408 22119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: No bootstrap required, opened a new log
I20260620 08:08:49.517838 22236 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:49.518601 22236 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:49.519603 22119 raft_consensus.cc:348] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.519835 22119 raft_consensus.cc:729] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a984f279252d4a1d802762e2b9b7de1d, State: Initialized, Role: FOLLOWER
I20260620 08:08:49.519992 22119 consensus_queue.cc:260] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } }
I20260620 08:08:49.520633 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54f840e1cab94e89affc1488bb854bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a984f279252d4a1d802762e2b9b7de1d"
I20260620 08:08:49.520750 22093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:49.521692 22093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f840e1cab94e89affc1488bb854bf4 in term 1.
I20260620 08:08:49.522209 22091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [term 1 FOLLOWER]: Refusing update from remote peer 54f840e1cab94e89affc1488bb854bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:49.522476 22119 sys_catalog.cc:564] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:49.522473 22236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4: Generated new TSK 0
I20260620 08:08:49.522480 22201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:49.522663 22236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:08:49.522897 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.523017 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:49.524845 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f840e1cab94e89affc1488bb854bf4. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.525022 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:49.525100 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:49.525197 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:49.525856 22258 mvcc.cc:205] Tried to move back new op lower bound from 7298838239318519808 to 7298838239156613120. Current Snapshot: MvccSnapshot[applied={T|T < 7298838239164792832}]
I20260620 08:08:49.528757 22261 catalog_manager.cc:1259] Loaded cluster ID: 181b4822dca241b2a00d601d5ba26029
I20260620 08:08:49.528873 22261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: loading cluster ID for follower catalog manager: success
I20260620 08:08:49.530138 22261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: acquiring CA information for follower catalog manager: success
I20260620 08:08:49.530552 22261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a984f279252d4a1d802762e2b9b7de1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 08:08:49.554602 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.552644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.31.115.252:46727)
0620 08:08:49.552711 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.552714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.553693 (+ 979us) server_negotiation.cc:407] Connection header received
0620 08:08:49.553842 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.553844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.553859 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.553901 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.554516 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47046: BlockingRecv error: recv got EOF from 127.0.0.1:47046 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 08:08:49.555588 22123 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.552316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.31.115.253:46825)
0620 08:08:49.552400 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.552403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.554924 (+ 2521us) server_negotiation.cc:407] Connection header received
0620 08:08:49.555055 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.555056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.555070 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.555109 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.555514 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46402: BlockingRecv error: recv got EOF from 127.0.0.1:46402 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 08:08:49.556612 22047 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.552976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.31.115.254:40013)
0620 08:08:49.553029 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.553031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.555886 (+ 2855us) server_negotiation.cc:407] Connection header received
0620 08:08:49.556003 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.556004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.556018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.556078 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.556523 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54058: BlockingRecv error: recv got EOF from 127.0.0.1:54058 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:08:49.572407 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.561920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.31.115.253:46825)
0620 08:08:49.561984 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.561986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.565786 (+ 3800us) server_negotiation.cc:407] Connection header received
0620 08:08:49.566051 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.566053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.566122 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.566471 (+ 349us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.568877 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.569762 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.571170 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.571575 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.571645 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.571989 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.572156 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.572257 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.572283 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.572306 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 08:08:49.586796 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.575328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.31.115.252:46727)
0620 08:08:49.575382 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.575385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.585945 (+ 10560us) server_negotiation.cc:407] Connection header received
0620 08:08:49.586115 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.586117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.586143 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.586198 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.586706 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47056: BlockingRecv error: recv got EOF from 127.0.0.1:47056 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:08:49.608515 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.598064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.31.115.253:46825)
0620 08:08:49.598115 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.598117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.598581 (+ 464us) server_negotiation.cc:407] Connection header received
0620 08:08:49.598682 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.598683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.598696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.598769 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.599071 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.606062 (+ 6991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.607654 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.607976 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.608019 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.608231 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.608269 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.608296 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.608398 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.608423 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 08:08:49.614998 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.614069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.31.115.252:46727)
0620 08:08:49.614142 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.614145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.614424 (+ 279us) server_negotiation.cc:407] Connection header received
0620 08:08:49.614514 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.614515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.614528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.614593 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.614923 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47068: BlockingRecv error: recv got EOF from 127.0.0.1:47068 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:08:49.626665 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.622355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.31.115.253:46825)
0620 08:08:49.622420 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.622423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.622651 (+ 228us) server_negotiation.cc:407] Connection header received
0620 08:08:49.623097 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.623099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.623113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.623144 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.623367 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.624278 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.625760 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.626114 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.626159 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.626382 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.626419 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.626446 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.626551 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.626573 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 08:08:49.635005 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.633979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.31.115.252:46727)
0620 08:08:49.634039 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.634041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.634391 (+ 350us) server_negotiation.cc:407] Connection header received
0620 08:08:49.634473 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.634475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.634490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.634569 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.634933 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47070: BlockingRecv error: recv got EOF from 127.0.0.1:47070 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:08:49.657022 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.652606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.31.115.253:46825)
0620 08:08:49.652658 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.652661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.653018 (+ 357us) server_negotiation.cc:407] Connection header received
0620 08:08:49.653132 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.653133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.653145 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.653176 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.653546 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.654550 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.656035 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.656365 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.656420 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.656685 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.656728 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.656762 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.656891 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.656917 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 08:08:49.662173 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.661023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.31.115.252:46727)
0620 08:08:49.661069 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.661071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.661371 (+ 300us) server_negotiation.cc:407] Connection header received
0620 08:08:49.661495 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.661497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.661510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.661550 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.662110 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47072: BlockingRecv error: recv got EOF from 127.0.0.1:47072 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:08:49.676364 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.671881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.31.115.253:46825)
0620 08:08:49.671932 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.671934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.672207 (+ 273us) server_negotiation.cc:407] Connection header received
0620 08:08:49.672320 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.672321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.672333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.672365 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.672586 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.673448 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.674911 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.675347 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.675661 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.675940 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.675980 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.676006 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.676135 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.676158 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 08:08:49.680615 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.679664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.31.115.252:46727)
0620 08:08:49.679711 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.679713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.680014 (+ 301us) server_negotiation.cc:407] Connection header received
0620 08:08:49.680120 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.680121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.680134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.680164 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.680553 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47086: BlockingRecv error: recv got EOF from 127.0.0.1:47086 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:08:49.701640 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.698147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.31.115.253:46825)
0620 08:08:49.698196 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.698199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.698528 (+ 329us) server_negotiation.cc:407] Connection header received
0620 08:08:49.698642 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.698644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.698655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.698685 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.698881 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.699619 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.700751 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.701049 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.701095 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.701310 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.701345 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.701385 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.701497 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.701520 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 08:08:49.705176 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.704265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.31.115.252:46727)
0620 08:08:49.704309 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.704311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.704592 (+ 281us) server_negotiation.cc:407] Connection header received
0620 08:08:49.704698 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.704699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.704711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.704741 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.705116 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47090: BlockingRecv error: recv got EOF from 127.0.0.1:47090 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:49.743845 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.739371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.31.115.253:46825)
0620 08:08:49.739456 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.739459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.739885 (+ 426us) server_negotiation.cc:407] Connection header received
0620 08:08:49.740510 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.740513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.740531 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.740565 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.740825 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.741781 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.743018 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.743311 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.743349 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.743564 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.743600 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.743625 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.743728 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.743753 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:08:49.747587 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.746742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.31.115.252:46727)
0620 08:08:49.746789 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.746791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.747010 (+ 219us) server_negotiation.cc:407] Connection header received
0620 08:08:49.747107 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.747108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.747119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.747151 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.747526 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47094: BlockingRecv error: recv got EOF from 127.0.0.1:47094 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:49.817803 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.813474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.31.115.253:46825)
0620 08:08:49.813827 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.813829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.813846 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:08:49.813888 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.813889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.813905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.814004 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.814409 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.814942 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.816426 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.816688 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.816782 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.817106 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.817154 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.817244 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.817497 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.817671 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
W20260620 08:08:49.821406 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.820194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.31.115.252:46727)
0620 08:08:49.820344 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.820348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.820671 (+ 323us) server_negotiation.cc:407] Connection header received
0620 08:08:49.820729 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.820731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.820748 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.820825 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.821310 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47110: BlockingRecv error: recv got EOF from 127.0.0.1:47110 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 08:08:49.954185 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:49.951041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.31.115.253:46825)
0620 08:08:49.951158 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.951161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.951174 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:08:49.951405 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.951406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.951428 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.951497 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:49.951669 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.952204 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.953050 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:49.953266 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:49.953365 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:49.953535 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:49.953599 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:49.953709 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:49.953988 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 08:08:49.954040 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 08:08:49.958999 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:49.957534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.31.115.252:46727)
0620 08:08:49.957638 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:49.957641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:49.958024 (+ 383us) server_negotiation.cc:407] Connection header received
0620 08:08:49.958063 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:49.958064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:49.958078 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:49.958234 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:49.958908 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47124: BlockingRecv error: recv got EOF from 127.0.0.1:47124 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:08:50.097429 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54f840e1cab94e89affc1488bb854bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f840e1cab94e89affc1488bb854bf4" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 40013 } } peers { permanent_uuid: "a984f279252d4a1d802762e2b9b7de1d" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 46825 } } peers { permanent_uuid: "3f0aca6f7dc94e8da2740558046b5307" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 46727 } } }
I20260620 08:08:50.097750 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f840e1cab94e89affc1488bb854bf4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:50.219326 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:50.216720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.31.115.253:46825)
0620 08:08:50.216807 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.216809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.216816 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:08:50.216844 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.216845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.216854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.216930 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:50.217089 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.217684 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.218533 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.218754 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.218842 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:50.219008 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:50.219047 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:50.219116 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:50.219187 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:08:50.219208 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 08:08:50.222500 22198 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:50.221779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.31.115.252:46727)
0620 08:08:50.221848 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.221850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.221863 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:08:50.221907 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.221908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.221921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.221991 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:50.222425 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47130: BlockingRecv error: recv got EOF from 127.0.0.1:47130 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:08:50.495810 22231 catalog_manager.cc:1259] Loaded cluster ID: 181b4822dca241b2a00d601d5ba26029
I20260620 08:08:50.495898 22231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: loading cluster ID for follower catalog manager: success
I20260620 08:08:50.496676 22231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: acquiring CA information for follower catalog manager: success
I20260620 08:08:50.496950 22231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f0aca6f7dc94e8da2740558046b5307: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:08:50.739563 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:50.736059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.31.115.253:46825)
0620 08:08:50.736270 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.736271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.736297 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:08:50.736457 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.736459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.736475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.736554 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:50.736774 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.737316 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.738254 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.738494 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.738562 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:50.739036 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:50.739071 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:50.739133 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:50.739335 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:08:50.739447 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260620 08:08:50.748417 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:50.742452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.31.115.252:46727)
0620 08:08:50.742605 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.742607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.743401 (+ 794us) server_negotiation.cc:407] Connection header received
0620 08:08:50.743437 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.743438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.743492 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.743588 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:50.743896 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.745400 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.747067 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.747498 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.747599 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:50.747893 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:50.748030 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:50.748146 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:50.748180 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 08:08:50.748211 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260620 08:08:50.754835 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:50.751322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.31.115.254:40013)
0620 08:08:50.751476 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.751478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.751489 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:08:50.751555 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.751556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.751750 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.751852 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:50.752199 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.752949 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.753794 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.754048 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.754114 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:50.754347 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:50.754467 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:50.754524 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:50.754718 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 08:08:50.754746 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":50,"threads_started":1}
I20260620 08:08:50.755003 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:50.751427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.31.115.252:46727)
0620 08:08:50.751496 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.751498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.751891 (+ 393us) server_negotiation.cc:407] Connection header received
0620 08:08:50.751935 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.751936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.751945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.752022 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:50.752173 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.752690 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.753989 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.754250 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.754335 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:50.754555 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:50.754607 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:50.754677 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:50.754878 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 08:08:50.754910 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:08:50.755498 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:50.751320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.31.115.253:46825)
0620 08:08:50.751378 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:50.751380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:50.751696 (+ 316us) server_negotiation.cc:407] Connection header received
0620 08:08:50.751740 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:50.751741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:50.751754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:50.751845 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:50.752041 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.752826 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.754375 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:50.754671 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:50.754738 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:50.754891 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:50.754928 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:50.754980 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:50.755388 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 08:08:50.755420 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:08:50.757443 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 21976
I20260620 08:08:50.766078 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 22051
I20260620 08:08:50.774930 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 22126
2026-06-20T08:08:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3138 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:08:50 dist-test-slave-c4w8 krb5kdc[22400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:08:50 dist-test-slave-c4w8 krb5kdc[22400](info): set up 2 sockets
Jun 20 08:08:50 dist-test-slave-c4w8 krb5kdc[22400](info): commencing operation
krb5kdc: starting...
W20260620 08:08:51.915364 32207 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/test-user.keytab.
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.115.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
Entry for principal kudu/127.31.115.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab.
I20260620 08:08:52.049952 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:46861
--webserver_interface=127.31.115.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:46861,127.31.115.253:41289,127.31.115.252:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:08:52.137329 22413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:08:52.137607 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:52.137655 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:52.137701 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:52.138945 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:52.139016 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:52.139042 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:52.139070 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:52.139097 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:52.140611 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:46861,127.31.115.253:41289,127.31.115.252:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.254.keytab
--principal=kudu/127.31.115.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.254:46861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.115.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:52.140898 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:52.141141 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:52.146481 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:52.146481 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:52.146586 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:52.146744 22413 server_base.cc:1034] running on GCE node
I20260620 08:08:52.147071 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:52.147336 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:52.148507 22413 hybrid_clock.cc:648] HybridClock initialized: now 1781942932148495 us; error 33 us; skew 500 ppm
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:52.151563 22413 init.cc:377] Logged in from keytab as kudu/127.31.115.254@KRBTEST.COM (short username kudu)
I20260620 08:08:52.152030 22413 webserver.cc:460] Webserver started at http://127.31.115.254:39863/ using document root <none> and password file <none>
I20260620 08:08:52.152338 22413 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:52.152390 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:52.152511 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:52.153777 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data/instance:
uuid: "331a4747693a46e0b51545c4d39dcc1c"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "1910e08fc4305d4f9d161dad2258450a"
server_key_iv: "62f77d37ac9d7c9e776e65e39ab1a8f0"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.154148 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal/instance:
uuid: "331a4747693a46e0b51545c4d39dcc1c"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "1910e08fc4305d4f9d161dad2258450a"
server_key_iv: "62f77d37ac9d7c9e776e65e39ab1a8f0"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.155674 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:52.156703 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:52.157101 22413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:52.157200 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal
uuid: "331a4747693a46e0b51545c4d39dcc1c"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "1910e08fc4305d4f9d161dad2258450a"
server_key_iv: "62f77d37ac9d7c9e776e65e39ab1a8f0"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.157368 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:52.182606 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:52.185227 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:52.185410 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:52.190371 22413 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.254:46861
I20260620 08:08:52.190414 22480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.254:46861 every 8 connection(s)
I20260620 08:08:52.190829 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/data/info.pb
I20260620 08:08:52.191809 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:52.193552 22481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.196095 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 22413
I20260620 08:08:52.196241 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-0/wal/instance
I20260620 08:08:52.196545 32207 external_mini_cluster.cc:1428] Setting key 333acaa5ee1a7765b73c378708726f20
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.115.253@KRBTEST.COM; defaulting to no policy
I20260620 08:08:52.205415 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.194409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:46861 (local address 127.0.0.1:55328)
0620 08:08:52.194611 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.194657 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.195148 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.195464 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.195466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.195747 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.195944 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.195949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.197270 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.197271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.198428 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.198432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.198988 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.198989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.199082 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.199646 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.199658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.201514 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.203648 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.203656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.203659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.203895 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.204172 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.204173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.204173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.204273 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.204725 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.204731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.204810 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.204930 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.205197 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":101,"thread_start_us":114,"threads_started":1}
I20260620 08:08:52.205415 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.194467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.31.115.254:46861)
0620 08:08:52.194657 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.194670 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.194788 (+ 118us) server_negotiation.cc:407] Connection header received
0620 08:08:52.195338 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.195340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.195361 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.195458 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.196058 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.197159 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.198531 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.198882 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.198989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.201647 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.201655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.201655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.201663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.203527 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.203999 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.204003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.204003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.204096 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.204358 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.204359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.204359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.204637 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.204748 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.204968 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.205124 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":120,"threads_started":1}
I20260620 08:08:52.206421 22481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.206913 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.206762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50528)
0620 08:08:52.206832 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.206887 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 08:08:52.207098 22431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.253@KRBTEST.COM" created.
I20260620 08:08:52.207756 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.207626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50542)
0620 08:08:52.207691 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.207726 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:08:52.207883 22481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:41289: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab.
Entry for principal kudu/127.31.115.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab.
I20260620 08:08:52.226060 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:41289
--webserver_interface=127.31.115.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab
--principal=kudu/127.31.115.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:46861,127.31.115.253:41289,127.31.115.252:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:08:52.257200 22481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } attempt: 1
I20260620 08:08:52.257882 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.257716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50548)
0620 08:08:52.257790 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.257849 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 08:08:52.258318 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.258225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50560)
0620 08:08:52.258271 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.258299 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 08:08:52.258428 22481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:41289: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
W20260620 08:08:52.315062 22488 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:08:52.315317 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:52.315342 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:52.315369 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:52.316704 22488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:52.316756 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:52.316773 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:52.316787 22488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:52.316812 22488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:52.318557 22488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:46861,127.31.115.253:41289,127.31.115.252:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.253.keytab
--principal=kudu/127.31.115.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.253:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.115.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:52.318804 22488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:52.319025 22488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:52.324582 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:52.324625 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:52.324617 22496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:52.325078 22488 server_base.cc:1034] running on GCE node
I20260620 08:08:52.325487 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:52.325892 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:52.326587 22481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } attempt: 2
I20260620 08:08:52.327093 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.326961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50562)
0620 08:08:52.327025 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.327063 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 08:08:52.327083 22488 hybrid_clock.cc:648] HybridClock initialized: now 1781942932327054 us; error 46 us; skew 500 ppm
I20260620 08:08:52.327448 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.327361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50576)
0620 08:08:52.327394 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.327421 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 08:08:52.327544 22481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.253:41289: Network error: Client connection negotiation failed: client connection to 127.31.115.253:41289: connect: Connection refused (error 111)
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:52.330278 22488 init.cc:377] Logged in from keytab as kudu/127.31.115.253@KRBTEST.COM (short username kudu)
I20260620 08:08:52.330771 22488 webserver.cc:460] Webserver started at http://127.31.115.253:41217/ using document root <none> and password file <none>
I20260620 08:08:52.331039 22488 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:52.331125 22488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:52.331271 22488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:52.332403 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data/instance:
uuid: "c346542df40b45f088bf0ebf95e6f55f"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "1e94486fb4a47e28aa94fea549ef43ed"
server_key_iv: "a312e53793158e506c07d2f14e22754d"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.332778 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal/instance:
uuid: "c346542df40b45f088bf0ebf95e6f55f"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "1e94486fb4a47e28aa94fea549ef43ed"
server_key_iv: "a312e53793158e506c07d2f14e22754d"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.334400 22488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:08:52.335289 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:52.335539 22488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:52.335637 22488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal
uuid: "c346542df40b45f088bf0ebf95e6f55f"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "1e94486fb4a47e28aa94fea549ef43ed"
server_key_iv: "a312e53793158e506c07d2f14e22754d"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.335702 22488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:52.348788 22488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:52.351250 22488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:52.351374 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:52.355784 22488 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.253:41289
I20260620 08:08:52.356235 22488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/data/info.pb
I20260620 08:08:52.356241 22555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.253:41289 every 8 connection(s)
I20260620 08:08:52.357023 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:52.358695 22556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.361835 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 22488
I20260620 08:08:52.361975 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-1/wal/instance
I20260620 08:08:52.362192 32207 external_mini_cluster.cc:1428] Setting key 34be62459e8e540280bed48f63c569c7
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
I20260620 08:08:52.369810 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.359530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.31.115.254:46861)
0620 08:08:52.359610 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.359613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.359839 (+ 226us) server_negotiation.cc:407] Connection header received
0620 08:08:52.360373 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.360374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.360390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.360443 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.361199 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.361848 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.362966 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.363182 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.363279 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.365994 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.365995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.365996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.366005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.368161 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.368719 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.368720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.368721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.368803 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.369070 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.369071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.369071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.369203 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.369276 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.369586 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.369696 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:08:52.370028 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.359499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:46861 (local address 127.0.0.1:55338)
0620 08:08:52.359690 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.359727 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.360257 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.360517 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.360522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.360910 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.361106 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.361110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.361940 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.361941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.362846 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.362848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.363285 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.363286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.363427 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.364012 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.364022 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.365876 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.368278 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.368295 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.368297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.368623 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.368877 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.368878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.368879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.369001 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.369280 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.369295 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.369446 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.369553 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.369770 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.31.115.252@KRBTEST.COM; defaulting to no policy
I20260620 08:08:52.371383 22556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.115.252@KRBTEST.COM" created.
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:52.380263 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.371717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50586)
0620 08:08:52.371889 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.371897 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.371931 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.372110 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.372112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.372261 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.372332 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.372334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.373461 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.373462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.374343 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.374346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.374977 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.374979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.375074 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.375617 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.375623 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.377099 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.379108 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.379109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.379110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.379283 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.379513 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.379514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.379514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.379570 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.379892 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.379893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.379933 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.380000 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.380115 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 08:08:52.380303 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.371764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.31.115.253:41289)
0620 08:08:52.371916 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.371920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.371942 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:08:52.371980 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.371981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.371994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.372054 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.372439 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.373336 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.374451 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.374858 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.374931 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.377212 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.377227 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.377227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.377239 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.379007 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.379381 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.379384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.379385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.379443 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.379642 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.379643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.379643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.379809 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.379903 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.380041 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.380171 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260620 08:08:52.381448 22556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.381932 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.381785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:48974)
0620 08:08:52.381841 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.381906 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:08:52.382139 22505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
I20260620 08:08:52.384022 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.383873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:48990)
0620 08:08:52.383943 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.383984 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 08:08:52.384138 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:34313: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.115.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab.
Entry for principal kudu/127.31.115.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab.
I20260620 08:08:52.394995 32207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvjmHcA/build/release/bin/kudu
/tmp/dist-test-taskvjmHcA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:34313
--webserver_interface=127.31.115.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab
--principal=kudu/127.31.115.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.115.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.115.254:46861,127.31.115.253:41289,127.31.115.252:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:08:52.418756 22481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } attempt: 3
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:52.427183 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.419212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50590)
0620 08:08:52.419291 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.419306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.419372 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.419538 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.419539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.419743 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.419819 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.419821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.420644 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.420645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.421466 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.421469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.421996 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.421997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.422075 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.422534 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.422540 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.424056 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.426003 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.426004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.426004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.426192 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.426476 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.426477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.426477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.426558 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.426895 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.426897 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.426944 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.427013 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.427098 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 08:08:52.427238 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.419322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.31.115.253:41289)
0620 08:08:52.419395 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.419397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.419410 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:08:52.419437 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.419437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.419449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.419504 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.419943 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.420554 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.421569 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.421917 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.421985 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.424146 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.424147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.424147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.424156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.425901 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.426294 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.426295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.426295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.426359 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.426643 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.426644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.426644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.426794 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.426862 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.427055 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.427145 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:08:52.427852 22481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.428203 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.428097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:48992)
0620 08:08:52.428147 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.428180 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 08:08:52.428574 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.428456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:48998)
0620 08:08:52.428520 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.428552 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 08:08:52.428719 22481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:34313: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
I20260620 08:08:52.433401 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } attempt: 1
I20260620 08:08:52.433917 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.433772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49008)
0620 08:08:52.433837 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.433879 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 08:08:52.434391 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.434247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49022)
0620 08:08:52.434330 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.434367 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 08:08:52.434538 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:34313: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
I20260620 08:08:52.459918 22481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } attempt: 1
I20260620 08:08:52.460479 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.460319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49028)
0620 08:08:52.460386 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.460439 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:08:52.460903 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.460803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49034)
0620 08:08:52.460839 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.460872 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 08:08:52.461095 22481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:34313: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
W20260620 08:08:52.489076 22563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:08:52.489337 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:08:52.489360 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:08:52.489385 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:08:52.490867 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:08:52.490919 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:08:52.490935 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:08:52.490950 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:08:52.490965 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:08:52.492689 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.115.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.115.254:46861,127.31.115.253:41289,127.31.115.252:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/krb5kdc/kudu_127.31.115.252.keytab
--principal=kudu/127.31.115.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.115.252:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.115.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:08:52.492993 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:08:52.493265 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:08:52.498893 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:52.498893 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:08:52.499004 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:08:52.499279 22563 server_base.cc:1034] running on GCE node
I20260620 08:08:52.499756 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:08:52.500191 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:08:52.501399 22563 hybrid_clock.cc:648] HybridClock initialized: now 1781942932501381 us; error 60 us; skew 500 ppm
I20260620 08:08:52.502705 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } attempt: 2
I20260620 08:08:52.503161 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.503024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49038)
0620 08:08:52.503091 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.503132 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 08:08:52.503566 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.503467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49050)
0620 08:08:52.503515 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.503541 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:08:52.503665 22556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.115.252:34313: Network error: Client connection negotiation failed: client connection to 127.31.115.252:34313: connect: Connection refused (error 111)
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:08:52.504362 22563 init.cc:377] Logged in from keytab as kudu/127.31.115.252@KRBTEST.COM (short username kudu)
I20260620 08:08:52.504822 22563 webserver.cc:460] Webserver started at http://127.31.115.252:44295/ using document root <none> and password file <none>
I20260620 08:08:52.505040 22563 fs_manager.cc:359] Metadata directory not provided
I20260620 08:08:52.505085 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:08:52.505188 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:08:52.506462 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data/instance:
uuid: "00565c2d2cc54615ac01b3d23e5772ef"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "e4e883d0938aac852599ebc84462c8c0"
server_key_iv: "a1df4627212702a8c6c634807d684dea"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.506821 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal/instance:
uuid: "00565c2d2cc54615ac01b3d23e5772ef"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "e4e883d0938aac852599ebc84462c8c0"
server_key_iv: "a1df4627212702a8c6c634807d684dea"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.508416 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:08:52.509256 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:08:52.509469 22563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:08:52.509586 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data,/tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal
uuid: "00565c2d2cc54615ac01b3d23e5772ef"
format_stamp: "Formatted at 2026-06-20 08:08:52 on dist-test-slave-c4w8"
server_key: "e4e883d0938aac852599ebc84462c8c0"
server_key_iv: "a1df4627212702a8c6c634807d684dea"
server_key_version: "encryptionkey@0"
I20260620 08:08:52.509750 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:08:52.523185 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:08:52.525864 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:08:52.526048 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:08:52.530589 22563 rpc_server.cc:307] RPC server started. Bound to: 127.31.115.252:34313
I20260620 08:08:52.530646 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.115.252:34313 every 8 connection(s)
I20260620 08:08:52.531081 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/data/info.pb
I20260620 08:08:52.532142 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:08:52.533982 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.536319 22481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } attempt: 2
I20260620 08:08:52.540989 32207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvjmHcA/build/release/bin/kudu as pid 22563
I20260620 08:08:52.541136 32207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvjmHcA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781942645567166-32207-0/minicluster-data/master-2/wal/instance
I20260620 08:08:52.541428 32207 external_mini_cluster.cc:1428] Setting key cec2a9fab9a086af0fb3c1e26e48e2ea
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.254@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:08:52.547039 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.534872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.31.115.254:46861)
0620 08:08:52.535060 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.535063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.535255 (+ 192us) server_negotiation.cc:407] Connection header received
0620 08:08:52.535976 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.535978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.535992 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.536043 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.536820 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.537588 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.539132 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.539440 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.539504 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.543077 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.543078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.543079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.543086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.544931 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.545472 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.545474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.545475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.545547 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.545901 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.545905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.545905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.546139 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.546203 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.546787 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.546888 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 08:08:52.547187 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.536675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49054)
0620 08:08:52.536746 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.536760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.536834 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.537086 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.537087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.537267 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.537307 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.537309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.538441 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.538442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.539308 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.539311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.539966 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.539968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.540039 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.540567 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.540572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.543074 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.545690 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.545691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.545691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.545923 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.546200 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.546201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.546202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.546285 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.546670 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.546671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.546730 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.546838 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.546957 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":4}
I20260620 08:08:52.547258 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.534723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.254:46861 (local address 127.0.0.1:55352)
0620 08:08:52.535120 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.535161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.535841 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.536121 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.536125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.536411 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.536717 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.536724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.537703 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.537704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.539034 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.539037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.539526 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.539527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.539660 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.540421 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.540431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.542957 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.545046 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.545056 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.545058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.545397 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.545652 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.545653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.545653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.545822 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.546229 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.546243 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.546616 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.546772 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.546973 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":215,"thread_start_us":152,"threads_started":1}
I20260620 08:08:52.547998 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:08:52.548520 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.536770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.31.115.252:34313)
0620 08:08:52.536932 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.536937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.536958 (+ 21us) server_negotiation.cc:407] Connection header received
0620 08:08:52.536977 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.536979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.536995 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.537058 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.537407 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.538346 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.539403 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.539827 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.539905 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.543179 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.543189 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.543190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.543198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.545598 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.546033 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.546036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.546037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.546097 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.546365 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.546366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.546367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.546594 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.546680 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.546877 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.547040 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:08:52.551517 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c: Bootstrap starting.
I20260620 08:08:52.552320 22481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:52.552623 22481 log.cc:824] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:52.553473 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c: No bootstrap required, opened a new log
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:52.554538 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.542117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.31.115.252:34313)
0620 08:08:52.542297 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.542299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.542471 (+ 172us) server_negotiation.cc:407] Connection header received
0620 08:08:52.542581 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.542582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.542593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.542683 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.543556 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.544294 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.545392 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.545744 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.545816 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.549985 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.549987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.549987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.549995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.552809 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.553434 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.553435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.553436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.553502 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.553768 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.553769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.553770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.553914 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.554070 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.554285 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.554373 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":118,"threads_started":1}
I20260620 08:08:52.556372 22481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.556643 22481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331a4747693a46e0b51545c4d39dcc1c, State: Initialized, Role: FOLLOWER
I20260620 08:08:52.556764 22481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.557150 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.557247 22481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:52.557439 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:52.557818 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.548314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.253:41289 (local address 127.0.0.1:50604)
0620 08:08:52.548372 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.548383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.548424 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.548599 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.548600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.548785 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.548827 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.548828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.550351 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.550352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.551213 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.551216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.551769 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.551770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.551847 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.552201 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.552207 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.553899 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.556409 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.556410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.556411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.556652 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.556960 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.556961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.556962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.557041 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.557406 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.557406 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.557482 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.557552 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.557713 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:08:52.558238 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.548396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.31.115.253:41289)
0620 08:08:52.548452 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.548454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.548476 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:08:52.548504 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.548505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.548518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.548576 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.548943 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.550182 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.551321 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.551675 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.551769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.554000 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.554002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.554002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.554010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.556251 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.556763 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.556764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.556764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.556853 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.557127 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.557128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.557128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.557295 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.557346 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.557969 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.558088 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:08:52.558813 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
W20260620 08:08:52.559774 22649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:52.559857 22649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.254@KRBTEST.COM
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.252@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:08:52.560704 22413 master_runner.cc:186] Error getting master registration for 127.31.115.253:41289: OK, instance_id { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" instance_seqno: 1781942932354483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:08:52.564072 22413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:08:52.565466 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.555639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.31.115.254:46861)
0620 08:08:52.555680 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.555683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.555691 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:08:52.555716 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.555716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.555726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.555794 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.556447 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.557213 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.558289 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.558589 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.558660 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.561496 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.561498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.561498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.561507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.563410 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.563823 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.563825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.563825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.563885 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.564216 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.564217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.564217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.564353 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.564784 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.565272 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.565359 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:52.567200 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.559093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49064)
0620 08:08:52.559207 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.559218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.559281 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.559438 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.559438 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.559645 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.559688 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.559691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.560674 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.560675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.561501 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.561503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.562132 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.562133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.562194 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.562654 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.562659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.564199 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.565920 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.565921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.565921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.566139 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.566421 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.566422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.566422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.566500 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.566862 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.566863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.566902 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.566970 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.567084 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 08:08:52.567580 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.559172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.31.115.252:34313)
0620 08:08:52.559220 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.559224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.559258 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:08:52.559348 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.559349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.559360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.559386 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.559781 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.560584 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.561601 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.561986 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.562043 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.564320 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.564322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.564322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.564337 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.565828 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.566240 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.566241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.566242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.566301 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.566600 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.566602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.566602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.566756 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.566837 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.567005 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.567492 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":25}
I20260620 08:08:52.569103 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: Bootstrap starting.
I20260620 08:08:52.569898 22631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:52.570221 22631 log.cc:824] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:52.571003 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: No bootstrap required, opened a new log
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.115.253@KRBTEST.COM
I20260620 08:08:52.572755 22631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.572937 22631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00565c2d2cc54615ac01b3d23e5772ef, State: Initialized, Role: FOLLOWER
I20260620 08:08:52.573030 22631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.573361 22631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:52.575137 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.575245 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:08:52.576141 22663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:52.576242 22663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:52.576253 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.566820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.31.115.253:41289)
0620 08:08:52.566862 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.566864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.566872 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:08:52.566897 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.566898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.566910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.566969 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.567628 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.568296 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.568974 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.569188 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.569272 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.572284 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.572285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.572286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.572295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.574293 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.574768 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.574770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.574770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.574839 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.575393 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.575395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.575396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.575602 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.575795 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.576046 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.576139 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:52.577127 22563 master_runner.cc:186] Error getting master registration for 127.31.115.253:41289: OK, instance_id { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" instance_seqno: 1781942932354483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:08:52.579715 22563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:08:52.580687 32207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:08:52.581877 22455 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "331a4747693a46e0b51545c4d39dcc1c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55364
I20260620 08:08:52.582070 22455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 08:08:52.582139 22455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:52.583447 22455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.583774 22455 leader_election.cc:288] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [CANDIDATE]: Term 1 election: Requested vote from peers c346542df40b45f088bf0ebf95e6f55f (127.31.115.253:41289), 00565c2d2cc54615ac01b3d23e5772ef (127.31.115.252:34313)
I20260620 08:08:52.584252 22605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "331a4747693a46e0b51545c4d39dcc1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00565c2d2cc54615ac01b3d23e5772ef"
I20260620 08:08:52.584393 22605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:52.585860 22605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331a4747693a46e0b51545c4d39dcc1c in term 1.
I20260620 08:08:52.586107 22430 leader_election.cc:302] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00565c2d2cc54615ac01b3d23e5772ef, 331a4747693a46e0b51545c4d39dcc1c; no voters:
I20260620 08:08:52.586288 22638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:08:52.586457 22638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [term 1 LEADER]: Becoming Leader. State: Replica: 331a4747693a46e0b51545c4d39dcc1c, State: Running, Role: LEADER
I20260620 08:08:52.586570 22638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.587069 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331a4747693a46e0b51545c4d39dcc1c. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.587157 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:52.587422 22672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:08:52.587632 22672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:08:52.589982 22605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [term 1 FOLLOWER]: Refusing update from remote peer 331a4747693a46e0b51545c4d39dcc1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:52.590355 22638 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:52.591743 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331a4747693a46e0b51545c4d39dcc1c. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.591858 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:52.593984 22673 mvcc.cc:205] Tried to move back new op lower bound from 7298838251885506560 to 7298838251875528704. Current Snapshot: MvccSnapshot[applied={T|T < 7298838251885506560}]
I20260620 08:08:52.594509 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.594606 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:52.594874 22556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } attempt: 3
I20260620 08:08:52.595280 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.595373 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:52.595677 22674 mvcc.cc:205] Tried to move back new op lower bound from 7298838251885506560 to 7298838251875528704. Current Snapshot: MvccSnapshot[applied={T|T < 7298838251885506560}]
I20260620 08:08:52.597584 22672 catalog_manager.cc:1347] Generated new cluster ID: d35e86dfa5d5434eafb86bfabccb1f66
I20260620 08:08:52.597707 22672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:08:52 dist-test-slave-c4w8 krb5kdc[22400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781942932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.115.253@KRBTEST.COM for kudu/127.31.115.252@KRBTEST.COM
I20260620 08:08:52.605382 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.585175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.31.115.253:41289)
0620 08:08:52.585236 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.585238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.585433 (+ 195us) server_negotiation.cc:407] Connection header received
0620 08:08:52.585567 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.585568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.585580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.585633 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.587583 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.588543 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.590653 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.591514 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.591566 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.597546 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.597548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.597548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.597556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.601031 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.602004 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.602005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.602005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.602065 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.602634 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.602635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.602635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.602786 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.602879 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.605190 (+ 2311us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.605278 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 08:08:52.612665 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.595318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.31.115.252:34313)
0620 08:08:52.595356 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.595358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.595365 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:08:52.595386 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.595387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.595398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.595444 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.595987 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.599146 (+ 3159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.601135 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.601497 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.601562 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.608020 (+ 6458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.608022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.608022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.608030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.609796 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.611604 (+ 1808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.611605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.611605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.611667 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.611885 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.611885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.611886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.612014 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.612074 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.612474 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.612563 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:08:52.613068 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.595112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.115.252:34313 (local address 127.0.0.1:49084)
0620 08:08:52.595187 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 08:08:52.595213 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 08:08:52.595250 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:08:52.595568 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:08:52.595569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:08:52.595777 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:08:52.595827 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.595829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.599253 (+ 3424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.599254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.600088 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:08:52.600091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.602133 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.602134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:08:52.602203 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.603784 (+ 1581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:08:52.603790 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:08:52.607920 (+ 4130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:08:52.609896 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.609897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.609898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.610114 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.611754 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:08:52.611755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:08:52.611755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:08:52.611820 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:08:52.612101 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:08:52.612102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:08:52.612151 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 08:08:52.612258 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:08:52.612370 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 08:08:52.615511 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: Bootstrap starting.
I20260620 08:08:52.616341 22556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: Neither blocks nor log segments found. Creating new log.
I20260620 08:08:52.616675 22556 log.cc:824] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:08:52.618692 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.594275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.31.115.252:34313)
0620 08:08:52.594313 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.594315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.603455 (+ 9140us) server_negotiation.cc:407] Connection header received
0620 08:08:52.603482 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.603483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.603495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.603544 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.606821 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.607560 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.611167 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.611489 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.611553 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.615212 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.615214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.615215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.615223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.617040 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.617560 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.617561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.617562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.617638 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.617888 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.617889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.617889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.618016 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.618101 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.618517 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.618593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:08:52.625491 22672 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:08:52.626376 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: No bootstrap required, opened a new log
I20260620 08:08:52.626560 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:52.593878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.31.115.254:46861)
0620 08:08:52.593933 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.593935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.594549 (+ 614us) server_negotiation.cc:407] Connection header received
0620 08:08:52.594790 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.594791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.594804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.594834 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.619763 (+ 24929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.620507 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.621556 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:52.621887 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:52.621932 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:52.623522 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:08:52.623524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:08:52.623524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:08:52.623532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:08:52.625198 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.625593 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.625594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.625594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.625670 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:08:52.625963 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:08:52.625964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:08:52.625964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:08:52.626093 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:08:52.626178 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:52.626386 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 08:08:52.626466 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:08:52.628683 22556 raft_consensus.cc:348] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.628862 22556 raft_consensus.cc:729] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c346542df40b45f088bf0ebf95e6f55f, State: Initialized, Role: FOLLOWER
I20260620 08:08:52.628952 22556 consensus_queue.cc:260] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } }
I20260620 08:08:52.629288 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.629405 22556 sys_catalog.cc:564] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:08:52.629480 22530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "331a4747693a46e0b51545c4d39dcc1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c346542df40b45f088bf0ebf95e6f55f"
I20260620 08:08:52.629601 22530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:08:52.630645 22530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 331a4747693a46e0b51545c4d39dcc1c in term 1.
I20260620 08:08:52.631069 22529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [term 1 FOLLOWER]: Refusing update from remote peer 331a4747693a46e0b51545c4d39dcc1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:08:52.631100 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:52.631639 22670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:08:52.633632 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331a4747693a46e0b51545c4d39dcc1c. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.633786 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:52.633865 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.633958 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:08:52.635607 22672 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:08:52.638571 22694 mvcc.cc:205] Tried to move back new op lower bound from 7298838252024782848 to 7298838251875528704. Current Snapshot: MvccSnapshot[applied={T|T < 7298838251885506560}]
W20260620 08:08:52.639863 22693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:08:52.639994 22693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:08:52.640208 22670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "331a4747693a46e0b51545c4d39dcc1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "331a4747693a46e0b51545c4d39dcc1c" member_type: VOTER last_known_addr { host: "127.31.115.254" port: 46861 } } peers { permanent_uuid: "c346542df40b45f088bf0ebf95e6f55f" member_type: VOTER last_known_addr { host: "127.31.115.253" port: 41289 } } peers { permanent_uuid: "00565c2d2cc54615ac01b3d23e5772ef" member_type: VOTER last_known_addr { host: "127.31.115.252" port: 34313 } } }
I20260620 08:08:52.640341 22670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c [sys.catalog]: This master's current role is: LEADER
I20260620 08:08:52.642306 22672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 331a4747693a46e0b51545c4d39dcc1c: Generated new TSK 0
I20260620 08:08:52.642503 22672 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:08:52.665472 22484 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.663768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.31.115.254:46861)
0620 08:08:52.663818 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.663820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.664760 (+ 940us) server_negotiation.cc:407] Connection header received
0620 08:08:52.664867 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.664868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.664880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.664911 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.665391 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55376: BlockingRecv error: recv got EOF from 127.0.0.1:55376 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 08:08:52.666443 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.663988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.31.115.253:41289)
0620 08:08:52.664027 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.664029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.665727 (+ 1698us) server_negotiation.cc:407] Connection header received
0620 08:08:52.665889 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.665890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.665903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.665935 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.666381 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50636: BlockingRecv error: recv got EOF from 127.0.0.1:50636 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 08:08:52.667080 22636 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.664366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.31.115.252:34313)
0620 08:08:52.664408 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.664410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.666026 (+ 1616us) server_negotiation.cc:407] Connection header received
0620 08:08:52.666660 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.666661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.666673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.666703 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.667021 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49090: BlockingRecv error: recv got EOF from 127.0.0.1:49090 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:08:52.669651 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.668777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.31.115.253:41289)
0620 08:08:52.668826 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.668828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.669046 (+ 218us) server_negotiation.cc:407] Connection header received
0620 08:08:52.669157 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.669158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.669179 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.669209 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.669564 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50642: BlockingRecv error: recv got EOF from 127.0.0.1:50642 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 08:08:52.673118 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.672181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.31.115.253:41289)
0620 08:08:52.672220 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.672222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.672539 (+ 317us) server_negotiation.cc:407] Connection header received
0620 08:08:52.672647 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.672648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.672660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.672689 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.673063 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50648: BlockingRecv error: recv got EOF from 127.0.0.1:50648 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:08:52.679329 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.678089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.31.115.253:41289)
0620 08:08:52.678148 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.678150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.678457 (+ 307us) server_negotiation.cc:407] Connection header received
0620 08:08:52.678569 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.678570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.678583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.678613 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.679264 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50662: BlockingRecv error: recv got EOF from 127.0.0.1:50662 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:08:52.685796 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.684795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.31.115.253:41289)
0620 08:08:52.684843 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.684845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.685133 (+ 288us) server_negotiation.cc:407] Connection header received
0620 08:08:52.685264 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.685266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.685278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.685309 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.685737 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50674: BlockingRecv error: recv got EOF from 127.0.0.1:50674 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 08:08:52.696450 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.695447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.31.115.253:41289)
0620 08:08:52.695507 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.695509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.695813 (+ 304us) server_negotiation.cc:407] Connection header received
0620 08:08:52.695929 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.695930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.695942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.695973 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.696386 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50676: BlockingRecv error: recv got EOF from 127.0.0.1:50676 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:08:52.715153 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.713928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.31.115.253:41289)
0620 08:08:52.713980 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.713982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.714475 (+ 493us) server_negotiation.cc:407] Connection header received
0620 08:08:52.714588 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.714589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.714602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.714642 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.715087 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50680: BlockingRecv error: recv got EOF from 127.0.0.1:50680 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 08:08:52.749532 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.748621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.31.115.253:41289)
0620 08:08:52.748668 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.748670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.748948 (+ 278us) server_negotiation.cc:407] Connection header received
0620 08:08:52.749054 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.749055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.749068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.749097 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.749470 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50696: BlockingRecv error: recv got EOF from 127.0.0.1:50696 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 08:08:52.815901 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.815100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.31.115.253:41289)
0620 08:08:52.815168 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.815171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.815264 (+ 93us) server_negotiation.cc:407] Connection header received
0620 08:08:52.815303 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.815304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.815318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.815405 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.815839 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50700: BlockingRecv error: recv got EOF from 127.0.0.1:50700 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 08:08:52.946632 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:52.945273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.31.115.253:41289)
0620 08:08:52.945401 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:52.945403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:52.945483 (+ 80us) server_negotiation.cc:407] Connection header received
0620 08:08:52.945584 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:52.945585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:52.945598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:52.945693 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:52.946549 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50708: BlockingRecv error: recv got EOF from 127.0.0.1:50708 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 08:08:53.205010 22560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:08:53.204301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.31.115.253:41289)
0620 08:08:53.204389 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:53.204391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:53.204404 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:08:53.204463 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:53.204464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:53.204479 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:53.204562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:08:53.204943 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50714: BlockingRecv error: recv got EOF from 127.0.0.1:50714 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:08:53.576960 22663 catalog_manager.cc:1259] Loaded cluster ID: d35e86dfa5d5434eafb86bfabccb1f66
I20260620 08:08:53.577042 22663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: loading cluster ID for follower catalog manager: success
I20260620 08:08:53.577953 22663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: acquiring CA information for follower catalog manager: success
I20260620 08:08:53.578239 22663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00565c2d2cc54615ac01b3d23e5772ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:08:53.640549 22693 catalog_manager.cc:1259] Loaded cluster ID: d35e86dfa5d5434eafb86bfabccb1f66
I20260620 08:08:53.640642 22693 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: loading cluster ID for follower catalog manager: success
I20260620 08:08:53.641486 22693 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: acquiring CA information for follower catalog manager: success
I20260620 08:08:53.641909 22693 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c346542df40b45f088bf0ebf95e6f55f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:08:53.722613 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:53.718400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.31.115.253:41289)
0620 08:08:53.718558 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:53.718561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:53.718667 (+ 106us) server_negotiation.cc:407] Connection header received
0620 08:08:53.718785 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:53.718786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:53.718874 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:53.718971 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:53.719213 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.720179 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.721480 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.721880 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.721965 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:53.722186 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:53.722323 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:53.722409 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:53.722471 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 08:08:53.722511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260620 08:08:53.728374 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:53.724888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.31.115.252:34313)
0620 08:08:53.725007 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:53.725009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:53.725167 (+ 158us) server_negotiation.cc:407] Connection header received
0620 08:08:53.725261 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:53.725262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:53.725317 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:53.725421 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:53.725561 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.726426 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.727282 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.727645 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.727725 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:53.727893 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:53.728053 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:53.728144 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:53.728227 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 08:08:53.728255 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260620 08:08:53.733886 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:53.731029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.31.115.252:34313)
0620 08:08:53.731225 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:53.731228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:53.731235 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:08:53.731317 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:53.731318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:53.731330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:53.731393 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:53.731557 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.732261 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.733110 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.733377 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.733443 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:53.733598 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:53.733659 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:53.733717 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:53.733768 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 08:08:53.733793 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260620 08:08:53.734258 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:53.730640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.31.115.253:41289)
0620 08:08:53.730693 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:53.730696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:53.730898 (+ 202us) server_negotiation.cc:407] Connection header received
0620 08:08:53.730971 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:53.730972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:53.730985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:53.731036 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:53.731290 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.732085 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.733355 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.733713 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.733773 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:53.733958 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:53.733993 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:53.734020 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:53.734121 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 08:08:53.734147 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:08:53.734432 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:08:53.730656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.31.115.254:46861)
0620 08:08:53.730781 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 08:08:53.730784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:08:53.730917 (+ 133us) server_negotiation.cc:407] Connection header received
0620 08:08:53.731053 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:08:53.731054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:08:53.731116 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:08:53.731182 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:08:53.731481 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.732281 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.733323 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:08:53.733679 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:08:53.733738 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:08:53.734008 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:08:53.734138 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:08:53.734197 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:08:53.734304 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 08:08:53.734336 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260620 08:08:53.736374 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 22413
I20260620 08:08:53.742854 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 22488
I20260620 08:08:53.748431 32207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvjmHcA/build/release/bin/kudu with pid 22563
2026-06-20T08:08:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2970 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6109 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288187 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST